In this paper, we demonstrate light field triangulation to determine depth distances and baselines in a plenoptic camera. Advances in micro lenses and image sensors have enabled plenoptic cameras to capture a scene from different viewpoints with sufficient spatial resolution. While object distances can be inferred from disparities in a stereo viewpoint pair using triangulation, this concept remains ambiguous when applied in the case of plenoptic cameras. We present a geometrical light field model allowing the triangulation to be applied to a plenoptic camera in order to predict object distances or specify baselines as desired. It is shown that distance estimates from our novel method match those of real objects placed in front of the camera. Additional benchmark tests with an optical design software further validate the model’s accuracy with deviations of less than ±0.33% for several main lens types and focus settings. A variety of applications in the automotive and robotics field can benefit from this estimation model.

Recent advances in camera technologies has led to the design of plenoptic cameras. This camera type can capture multiple images of the same scene using arrays of microlenses, where each microlens has a shifted location providing a separate view of the scene. Such a design results in a superior performance as compared to traditional cameras, enabling multi-view or multi-focal imaging captured in a single shot. However, currently available plenoptic cameras are limited in spatial resolution, which makes it difficult to use them in applications where sharpness or high resolution is key, such as the film industry. Our paper presents a novel light field hybrid super-resolution method that combines two classical super-resolution techniques for efficient application to plenoptic images. After this combination, we first segment the output hybrid super-resolution image into the objects of interest. Afterward, we apply sparse representation to super resolve the segmented image. This technique helps to improve the quality by decrease computations for light field images and extract significant features from the objects of interest. We demonstrate the gain achieved by the novel method as compared to the current relevant approaches in terms of both PSNR and SSIM for various enhanced spatial resolutions.

Virtual view synthesis is a key component of multi-view imaging systems that enable visual immersion environments for emerging applications, e.g., virtual reality and 360-degree video. Using a small collection of captured reference view-points, this technique reconstructs any view of a remote scene of interest navigated by a user, to enhance the perceived immersion experience. We carry out a convexity characterization analysis of the virtual view reconstruction error that is caused by compression of the captured multi-view content. This error is expressed as a function of the virtual viewpoint coordinate relative to the captured reference viewpoints. We derive fundamental insights about the nature of this dependency and formulate a prediction framework that is able to accurately predict the specific dependency shape, convex or concave, for given reference views, multi-view content and compression settings. We are able to integrate our analysis into a proof-of-concept coding framework and demonstrate considerable benefits over a baseline approach.

We study multicast of multi-view content in the video plus depth format to heterogeneous clients. We design a joint source-channel coding scheme based on view and rate embedded source coding and rateless channel coding. It comprises an optimization framework for joint view selection and source-channel rate allocation, and includes a fast method for separate optimization of the source and channel coding components, at a negligible performance loss wrt the joint solution. We demonstrate performance gains over a state-of-the-art method based on H.264/SVC, in the case of two client classes.

In dynamic spectrum access networks, cognitive radio terminals monitor their spectral environment in order to detect and opportunistically access unoccupied frequency channels. The overall performance of such networks depends on the spectrum occupancy or availability patterns. Accurate knowledge on the channel availability enables optimum performance of such networks in terms of spectrum and energy efficiency. This work proposes a novel probabilistic channel availability model that can describe the channel availability in different polarizations for mobile cognitive radio terminals that are likely to change their orientation during their operation. A Gaussian approximation is used to model the empirical occupancy data that was obtained through a measurement campaign in the cellular frequency bands within a realistic operational scenario.

Spectrum sensing is the cornerstone of cognitive radio technology and refers to the process of obtaining awareness of the radio spectrum usage in order to detect the presence of other users. Spectrum sensing algorithms consume considerable energy and time. Prediction methods for inferring the channel occupancy of future time instants have been proposed as a means of improving performance in terms of energy and time consumption. This paper studies the performance of a hidden Markov model (HMM) spectrum occupancy predictor as well as the improvement in sensing energy and time consumption based on real occupancy data obtained in the 2.4GHz ISM band. Experimental results show that the HMM-based occupancy predictor outperforms a kth order Markov and a 1-nearest neighbour (1NN) predictor. Our study also suggests that by employing such a predictive scheme in spectrum sensing, an improvement of up to 66% can be achieved in the required sensing energy and time.

We derive an optimization framework for joint view and rate scalable coding of multi-view video content represented in the texture plus depth format. The optimization enables the sender to select the subset of coded views and their encoding rates such that the aggregate distortion over a continuum of synthesized views is minimized. We construct the view and rate embedded bitstream such that it delivers optimal performance simultaneously over a discrete set of transmission rates. In conjunction, we develop a user interaction model that characterizes the view selection actions of the client as a Markov chain over a discrete state-space. We exploit the model within the context of our optimization to compute user-action-driven coding strategies that aim at enhancing the client's performance in terms of latency and video quality. Our optimization outperforms the state-of-the-art H.264 SVC codec as well as a multi-view wavelet-based coder equipped with a uniform rate allocation strategy, across all scenarios studied in our experiments. Equally important, we can achieve an arbitrarily fine granularity of encoding bit rates, while providing a novel functionality of view embedded encoding, unlike the other encoding methods that we examined. Finally, we observe that the interactivity-aware coding delivers superior performance over conventional allocation techniques that do not anticipate the client's view selection actions in their operation.

In this paper, we present a novel wavelet-based compression algorithm for multiview images. This method uses a layer-based representation, where the 3-D scene is approximated by a set of depth planes with their associated constant disparities. The layers are extracted from a collection of images captured at multiple viewpoints and transformed using the 3-D discrete wavelet transform (DWT). The DWT consists of the 1-D disparity compensated DWT across the viewpoints and the 2-D shape-adaptive DWT across the spatial dimensions. Finally, the wavelet coefficients are quantized and entropy coded along with the layer contours. To improve the rate-distortion performance of the entire coding method, we develop a bit allocation strategy for the distribution of the available bit budget between encoding the layer contours and the wavelet coefficients. The achieved performance of our proposed scheme outperforms the state-of-the-art codecs for several data sets of varying complexity.

We propose a novel multiview compression method for multiview images. The algorithm supports random access for interactive applications and has low storage requirements. The fundamental component of the method is the layer-based representation, which partitions the dataset into redundant layers characterized by a constant depth value. We exploit the redundant property of each layer and remove the side information uncertainty using Distributed Source Coding (DSC) principles. In comparison to independent coding, our method achieves a PSNR improvement of 3dB. Furthermore, we present a rate-distortion (RD) analysis which demonstrates that the proposed algorithm can achieve a better performance in comparison to independent coding.

Automatic annotation of video clips is desired to efficiently thumbnail user generated content available in the Internet. Automatic techniques typically focus on a selected set of inherent video features (such as scene-cut or shot boundaries) that are deemed to be salient. Along this line, the presence of camera flash light illumination is a feature of interest. This event is usually triggered manually (by a photographer) within a scene and the selected frame(s) they occur in are deemed to be interesting in the recording: for instance, the appearance of a celebrity in a party. In this paper we present a method to detect video frames that contain flashes originating from still cameras in user generated video clips. We focus on designing features for flash illumination detection using various measures of luminance change within video sequences. Using the proposed method, we obtain detection performance of approximately 89% which is 8% absolute improvement over the baseline method that uses only change in average illumination. We also illustrate a case where flashes are automatically detected in video clips of social gatherings that can be used for thumbnailing and browsing.

“Texture-plus-depth” has become a popular coding format for multiview image compression, where a decoder can synthesize images at intermediate viewpoints using encoded texture and depth maps of closest captured view locations via depth-image-based rendering (DIBR). As in other resource-constrained scenarios, limited available bits must be optimally distributed among captured texture and depth maps to minimize the expected signal distortion at the decoder. A specific challenge of multiview image compression for DIBR is that the encoder must allocate bits without the knowledge of how many and which specific virtual views will be synthesized at the decoder for viewing. In this paper, we derive a cubic synthesized view distortion model to describe the visual quality of an interpolated view as a function of the view's location. Given the model, one can easily find the virtual view location between two coded views where the maximum synthesized distortion occurs. Using a multiview image codec based on shape-adaptive wavelet transform, we show how optimal bit allocation can be performed to minimize the maximum view synthesis distortion at any intermediate viewpoint. Our experimental results show that the optimal bit allocation can outperform a common uniform bit allocation scheme by up to 1.0dB in coding efficiency performance, while simultaneously being competitive to a state-of-the-art H.264 codec.

The encoding of both texture and depth maps of multiview images, captured by a set of spatially correlated cameras, is important for any 3-D visual communication system based on depth-image-based rendering (DIBR). In this paper, we address the problem of efficient bit allocation among texture and depth maps of multiview images. More specifically, suppose we are given a coding tool to encode texture and depth maps at the encoder and a view-synthesis tool to construct intermediate views at the decoder using neighboring encoded texture and depth maps. Our goal is to determine how to best select captured views for encoding and distribute available bits among texture and depth maps of selected coded views, such that the visual distortion of desired constructed views is minimized. First, in order to obtain at the encoder a low complexity estimate of the visual quality of a large number of desired synthesized views, we derive a cubic distortion model based on basic DIBR properties, whose parameters are obtained using only a small number of viewpoint samples. Then, we demonstrate that the optimal selection of coded views and quantization levels for corresponding texture and depth maps is equivalent to the shortest path in a specially constructed 3-D trellis. Finally, we show that, using the assumptions of monotonicity in the predictor's quantization level and distance, suboptimal solutions can be efficiently pruned from the feasible space during solution search. Experiments show that our proposed efficient selection of coded views and quantization levels for corresponding texture and depth maps outperforms an alternative scheme using constant quantization levels for all maps (commonly used in video standard implementations) by up to 1.5 dB. Moreover, the complexity of our scheme can be reduced by at least 80% over the full solution search.

The encoding of both texture and depth maps of a set of multi-view images, captured by a set of spatially correlated cameras, is important for any 3D visual communication systems based on depth-image-based rendering (DIBR). In this paper, we address the problem of efficient bit allocation among texture and depth maps of multi-view images. We pose the following question: for chosen (1) coding tool to encode texture and depth maps at the encoder and (2) view synthesis tool to reconstruct uncoded views at the decoder, how to best select captured views for encoding and distribute available bits among texture and depth maps of selected coded views, such that visual distortion of a “metric” of reconstructed views is minimized. We show that using the monotonicity assumption, suboptimal solutions can be efficiently pruned from the feasible space during parameter search. Our experiments show that optimal selection of coded views and associated quantization levels for texture and depth maps can outperform a heuristic scheme using constant levels for all maps (commonly used in the standard implementations) by up to 2.0dB. Moreover, the complexity of our scheme can be reduced by up to 66% over full search without loss of optimality.

We propose a novel compression method for multiview images. The algorithm exploits the layer-based representation, which partitions the data set into planar layers characterized by a constant depth value. For efficient compression, the partitioned data is decorrelated using the separable three-dimensional wavelet transform across the viewpoint and spatial dimensions. The transform is modified to efficiently deal with occlusions and disparity variations for different depths. The generated transform coefficients are entropy coded. Experimental results show that our coding method is capable of outperforming the state-of-the-art algorithms, like H.264/AVC, for different data sets.

Novel coding tools have been proposed recently to encode texture and depth maps of multiview images, exploiting inter-view correlations, for depth-image-based rendering (DIBR). However, the important associated bit allocation problem for DIBR remains open: for chosen view coding and synthesis tools, how to allocate bits among texture and depth maps across encoded views, so that the fidelity of a set of V views reconstructed at the decoder is maximized, for a fixed bitrate budget? In this paper, we present an optimization strategy to select subset of texture and depth maps of the original V views for encoding at appropriate quantization levels, so that at the decoder, the combined quality of decoded views (using encoded texture maps) and synthesized views (using encoded texture and depth maps of neighboring views) is maximized. We show that using the monotonicity property, complexity of our strategy can be greatly reduced. Experiments show that using our strategy, one can achieve up to 0.83dB gain in PSNR improvement over a heuristic scheme of encoding only texture maps of all V views at constant quantization levels. Further, computation can be reduced by up to 66% over a full parameter search approach.

We consider the scenario of view synthesis via depth-image based rendering in multi-view imaging. We formulate a resource allocation problem of jointly assigning an optimal number of bits to compressed texture and depth images such that the maximum distortion of a synthesized view over a continuum of viewpoints between two encoded reference views is minimized, for a given bit budget. We construct simple yet accurate image models that characterize the pixel values at similar depths as first-order Gaussian auto-regressive processes. Based on our models, we derive an optimization procedure that numerically solves the formulated min-max problem using Lagrange relaxation. Through simulations we show that, for two captured views scenario, our optimization provides a significant gain (up to 2dB) in quality of the synthesized views for the same overall bit rate over a heuristic quantization that selects only two quantizers — one for the encoded texture images and the other for the depth images.

We present a novel and efficient multi-view depth map enhancement method proposed as a post-processing of initially estimated depth maps. The proposed method is based on edge, motion and scene depth-range adaptive median filtering and allows for an improved quality of virtual view synthesis. To enforce the spatial, temporal and inter-view coherence in the multi-view depth maps, the median filtering is applied to 4-dimensional windows that consist of the spatially neighboring depth map values taken at different viewpoints and time instants. A fast iterative block segmentation approach is adopted to adaptively shrink these windows in the presence of edges and motion for preservation of sharpness and realistic rendering and for improvement of the compression efficiency. We show that our enhancement method leads to a reduction of the coding bit-rate required for representation of the depth maps and also leads to a gain in the quality of synthesized views at arbitrary virtual viewpoints.

Iris recognition recently became an active field of biometric security because of reliability and easy non-invasive acquisition of the data. The randomness and stability of the iris textures allow for a convenient application in personal authentication and identification. In a novel iris recognition method presented here, the iris features are extracted using the oriented separable wavelet transforms (directionlets) and they are compared in terms of a weighted Hamming distance. The feature extraction and comparison are shift, size and rotation-invariant to the location of iris in the acquired image. The generated iris code is binary, whose length is fixed (and therefore commensurable), independent of the iris image, and comparatively short. The novel method shows a good performance when applied to a large database of irises and provides reliable identification and verification. At the same time, it preserves conceptual and computational simplicity and allows for a quick analysis and comparison of iris samples.

We present a novel multi-view depth map enhancement method deployed as a post-processing of initially estimated depth maps, which are incoherent in the temporal and inter-view dimensions. The proposed method is based on edge and motion-adaptive median filtering and allows for an improved quality of virtual view synthesis. To enforce the spatial, temporal and inter-view coherence in the multiview depth maps, the median filtering is applied to 4-dimensional windows that consist of the spatially neighbor depth map values taken at different viewpoints and time instants. These windows have locally adaptive shapes in a presence of edges or motion to preserve sharpness and realistic rendering. We show that our enhancement method leads to a reduction of a coding bit-rate required for representation of the depth maps and also to a gain in the quality of synthesized views at an arbitrary virtual viewpoint. At the same time, the method carries a low additional computational complexity.

A novel iris recognition method is presented. In the method, the iris features are extracted using the oriented separable wavelet transforms (directionlets) and they are compared in terms of a weighted Hamming distance. The feature extraction and comparison are shift-, size-, and rotation-invariant to the location of iris in the acquired image. The generated iris code isbinary, whose length is fixed (and therefore commensurable), independent of the iris image, and comparatively short. The novel method shows a good performance when applied to a large database of irises and provides reliable identification and verification. At the same time, it preserves conceptual and computational simplicity and allows for a quick analysis and comparison of iris samples.

We present a novel directionally adaptive image interpolation based on a multiple-direction wavelet transform, called directionlets. The algorithm uses directionlets to efficiently capture directional features and to extract edge information along different directions from the low-resolution image. Then, the high-resolution image is generated using this information to preserve sharpness of details. Our interpolation algorithm outperforms the state-of-the-art methods in terms of both numeric and visual quality of the interpolated image.

A novel directionally adaptive image resolution enhancement method is proposed. The method uses a multiple-direction wavelet transform, called directionlets, to efficiently extract edge information along different directions, not necessarily horizontal or vertical, from the low-resolution image. Then, the high-resolution image is synthesized using the extracted information to preserve sharpness of edges and texture. The novel algorithm provides the interpolated images at a higher resolution that are better than the images obtained by the state-of-the-art methods in terms of both numeric and visual quality.

In our previous work [1], we proposed a construction of critically sampled perfect reconstruction transforms with directional vanishing moments (DVMs) imposed in the corresponding basis functions along different directions, called directionlets. Here, we combine the directionlets with the space-frequency quantization (SFQ) image compression method, originally based on the standard two-dimensional (2-D) wavelet transform (WT) and proposed in [2]. We show that our new compression method outperforms the standard SFQ as well as the state-of-the-art compression methods, like SPIHT and JPEG-2000, in terms of the quality of compressed images, especially in a low-rate compression regime. We also show that the order of computational complexity remains the same, as compared to the complexity of the standard SFQ algorithm.

The standard separable 2-D wavelet transform (WT) has recently achieved a great success in image processing because it provides a sparse representation of smooth images. However, it fails to efficiently capture 1-D discontinuities, like edges or contours. These features, being elongated and characterized by geometrical regularity along different directions, intersect and generate many large magnitude wavelet coefficients. Since contours are very important elements in the visual perception of images, to provide a good visual quality of compressed images, it is fundamental to preserve good reconstruction of these directional features. In our previous work, we proposed a construction of critically sampled perfect reconstruction transforms with directional vanishing moments imposed in the corresponding basis functions along different directions, called directionlets. In this paper, we show how to design and implement a novel efficient space-frequency quantization (SFQ) compression algorithm using directionlets. Our new compression method outperforms the standard SFQ in a rate-distortion sense, both in terms of mean-square error and visual quality, especially in the low-rate compression regime. We also show that our compression method, does not increase the order of computational complexity as compared to the standard SFQ algorithm.

The standard separable two-dimensional (2-D) wavelet transform (WT) has recently achieved a great success in image processing because it provides a sparse representation of smooth images. However, it fails to capture efficiently one-dimensional (1-D) discontinuities, like edges and contours, that are anisotropic and characterized by geometrical regularity along different directions. In our previous work, we proposed a construction of critically sampled perfect reconstruction anisotropic transform with directional vanishing moments (DVM) imposed in the corresponding basis functions, called directionlets. Here, we show that the computational complexity of our transform is comparable to the complexity of the standard 2-D WT and substantially lower than the complexity of other similar approaches. We also present a zerotree-based image compression algorithm using directionlets that strongly outperforms the corresponding method based on the standard wavelets at low bit rates.

In spite of the success of the standard wavelet transform (WT) in image processing in recent years, the efficiency of its representation is limited by the spatial isotropy of its basis functions built in the horizontal and vertical directions. One-dimensional (1-D) discontinuities in images (edges and contours) that are very important elements in visual perception, intersect too many wavelet basis functions and lead to a nonsparse representation. To efficiently capture these anisotropic geometrical structures characterized by many more than the horizontal and vertical directions, a more complex multidirectional (M-DIR) and anisotropic transform is required. We present a new lattice-based perfect reconstruction and critically sampled anisotropic M-DIR WT. The transform retains the separable filtering and subsampling and the simplicity of computations and filter design from the standard two-dimensional WT, unlike in the case of some other directional transform constructions (e.g., curvelets, contourlets, or edgelets). The corresponding anisotropic basis unctions (directionlets) have directional vanishing moments along any two directions with rational slopes. Furthermore, we show that this novel transform provides an efficient tool for nonlinear approximation of images, achieving the approximation power O(N(-1.55)), which, while slower than the optimal rate O(N(-2)), is much better than O(N(-1)) achieved with wavelets, but at similar complexity.

In spite of the success of the standard wavelet transform (WT) in image processing, the efficiency of its representation is limited by the spatial isotropy of its basis functions built in only horizontal and vertical directions. One-dimensional (1-D) discontinuities in images (edges and contours), which are very important elements in visual perception, intersect too many wavelet basis functions and reduce the sparsity of the representation. To capture efficiently these anisotropic geometrical structures, a more complex multi-directional (M-DIR) and anisotropic transform is required. We present a new lattice-based perfect reconstruction and critically sampled anisotropic M-DIR WT (with the corresponding basis functions called directionlets) that retains the separable filtering and simple filter design from the standard two-dimensional (2-D) WT and imposes directional vanishing moments (DVM). Further-more, we show that this novel transform has non-linear approximation efficiency competitive to the other previously proposed over-sampled transform constructions.

The application of the wavelet transform in image processing is most frequently based on a separable construction. While simple, such an approach is not capable of capturing properly all 2D properties in images. In this paper, a new truly separable multidirectional transform is proposed with a subsampling method based on lattice theory. Applications are possible in many areas of image processing. Some promising improvements are achieved in nonlinear approximation and denoising of images.

The application of the wavelet transform in image processing is most frequently based on a separable transform. Lines and columns in an image are treated independently and the basis functions are simply products of corresponding one-dimensional functions. Such a method keeps simplicity in design and computation. A new two-dimensional approach is proposed, which retains the simplicity of separable processing, but allows more directionalities. The method can be applied in many areas like denoising, nonlinear approximation and compression. The results on nonlinear approximation and denoising show interesting gains compared to the standard two-dimensional analysis.