The performance of automated classification algorithms for medical images needs to be very high and especially robust in order to be adopted into healthcare. Most of the time the main challenge is unregistered data, since it is usually captured: 1) from different patients, 2) with different devices, and 3) at different time. Registration and normalization of the captured data is a necessary condition for success. In this paper we present for the first time an automated method to register eardrums from light-field data. This procedure uses the shape information captured by a light-field otoscope and compensates for the natural tilt of the eardrum, its size, and the camera viewpoint. Results on clinical data show that the proposed algorithm is robust and works well for different types of ear conditions.

Bulging is a medical characteristic of the eardrum that is crucial for the diagnosis of acute otitis media. This work proposes a novel classification method for distinguishing bulged eardrums from non-bulged ones. The method uses novel key features extracted from 3D data of the tympanic membrane, captured using a new type of otoscope, the light-field otoscope, capable of non-invasive 3D imaging of the middle ear. We first introduce a variety of geometrical and statistical descriptors (based on isocontours), and then select the most discriminative ones. Results on clinical data show that, when using the proposed feature descriptors, eardrum bulging can be automatically detected with an average accuracy of approximately 82%.

Near-eye multi-focal (light field) displays are known to be more advantageous than conventional stereoscopic 3D displays, because they can alleviate the vergence-accommodation conflict. While stereoscopic 3D displays are restricted to projecting images on a single focal plane, near-eye light field displays can form multiple focal planes and volumetrically render 3D data. Most existing near-eye light field displays use simplistic, uniformly spaced focal plane configuration (in dioptric space). In this paper, we present a novel technique that optimizes the focal plane configuration based on characteristics of the content to be rendered. We show that this technique can significantly improve the perceived visual quality of content visualized on such displays.

We present a study of depth-disparity calibration for augmented reality applications using binocular optical see-through displays. Two techniques were proposed and compared. The "paired-eyes" technique leverages the Panum's fusional area to help viewer find alignment between the virtual and physical objects. The "separate-eyes" technique eliminates the need of binocular fusion and involves using both eyes sequentially to check the virtual-physical object alignment on retinal images. We conducted a user study to measure the calibration results and assess the subjective experience of users with the proposed techniques.

We consider the calibration problem in plenoptic systems with the goal to estimate the system matrix given a set of known calibration patterns and their corresponding observations. The proposed calibration method is divided into a subspace identification step and a parameter estimation step. Exploiting newly discovered properties of the simulated system matrix, such as its low rank and the equality of its row and column spaces, we propose to identify the requisite subspace by computing the dominant left singular vectors of the observation matrix. Subsequently, we project each row of the observation matrix onto the subspace determined by these singular vectors. Our simulation results show that our method is able to accurately estimate the system matrix and significantly outperform a state-of-the-art low-rank recovery algorithm.

We present a method for 3D keypoint detection from light field data, typically obtained by planar camera arrays or plenoptic cameras. The proposed approach is based on construction of novel light field scale-depth spaces that are designed to leverage the specific properties of light fields. The constructed scale-depth spaces are based on a modified Gaussian kernel that is parametrized both in terms of scale of objects recorded by the light field and in terms of objects' depth. We prove theoretically that the new scale-depth space formulation and its spatial derivative satisfy the scale invariance property for all depths. By finding local extrema in such scale-depth spaces we locate 3D keypoints (such as 3D edges) and show that they outperform SURF keypoints on a 3D structure estimation task, both in accuracy and computational efficiency.

This paper presents a method for learning overcomplete dictionaries of atoms composed of two modalities that describe a 3D scene: 1) image intensity and 2) scene depth. We propose a novel joint basis pursuit (JBP) algorithm that finds related sparse features in two modalities using conic programming and we integrate it into a two-step dictionary learning algorithm. The JBP differs from related convex algorithms because it finds joint sparsity models with different atoms and different coefficient values for intensity and depth. This is crucial for recovering generative models where the same sparse underlying causes (3D features) give rise to different signals (intensity and depth). We give a bound for recovery error of sparse coefficients obtained by JBP, and show numerically that JBP is superior to the group lasso algorithm. When applied to the Middlebury depth-intensity database, our learning algorithm converges to a set of related features, such as pairs of depth and intensity edges or image textures and depth slants. Finally, we show that JBP outperforms state of the art methods on depth inpainting for time-of-flight and Microsoft Kinect 3D data.

We propose a method for object reconstruction from images obtained by a plenoptic camera. Our approach exploits a plenoptic system model based on diffraction analysis in order to formulate an inverse problem for object reconstruction. To solve this inverse problem, we propose a dictionary learning algorithm for signal reconstruction from measurements obtained by a deterministic linear system. In contrast to prior work in Compressive Sensing, we do not impose constraints on the measurement matrix, but allow it to be defined by the properties of a specified camera system. Given the measurement matrix, the proposed algorithm learns a dictionary from a large database of examples and simultaneously minimizes the mutual coherence between the measurement matrix and the dictionary. We evaluate the performance of the algorithm on object reconstruction from plenoptic system measurements and show that it outperforms existing solutions.

We present a new method for estimating 3D scene layers from light field data obtained by plenoptic cameras or camera arrays. The proposed method is based on a novel sparse generative model for light fields, which uses a dictionary of ray-like functions and combines them in a non-linear way via a set of occlusion masks. By estimating the set of sparse coefficients and masks, our method divides the light field into layers corresponding to different depths in a 3D scene, while taking occlusions into account. The proposed method can thus be used for 3D scene segmentation and in a myriad of inverse problems in light field imaging, such as view interpolation, denoising, inpainting and super-resolution.

We propose a new method for modeling the temporal correlation in videos, based on local transforms realized by Lie group operators. A large class of transforms can be theoretically described by these operators, however, we propose to learn from natural movies a subset of transforms that are statistically relevant for video representation. The proposed transformation modeling is further exploited to remove inter-view redundancy, i.e., as the prediction step of video encoding. Since the Lie group transformation coefficients are continuous, a quantization step is necessary for each transform. Therefore, we derive theoretical bounds on the distortion due to coefficient quantization. The experimental results demonstrate that the new prediction method with learned transforms leads to better rate-distortion performance at higher bit-rates, and competitive performance at lower bit-rates, compared to the standard prediction based on block-based motion estimation.

This paper proposes a framework for stereo image coding with effective representation of geometry in 3D scenes. We propose a joint sparse approximation framework for pairs of perspective images that are represented as linear expansions of atoms selected from a dictionary of geometric functions learned on a database of stereo perspective images. We then present a coding solution where atoms are selected iteratively as a trade-off between distortion and consistency of the geometry information. Experimental results on stereo images from the Middlebury database show that the new coder achieves better rate-distortion performance compared to the MPEG4-part10 scheme, at all rates. In addition to good rate-distortion performance, our flexible framework permits to build consistent image representations that capture the geometry of the scene. It certainly represents a promising solution towards the design of multi-view coding algorithms where the compressed stream inherently contains rich information about 3D geometry.

One of the major challenges in multi-view imaging is the definition of a representation that reveals the intrinsic geometry of the visual information. Sparse image representations with overcomplete geometric dictionaries offer a way to efficiently approximate these images, such that the multi-view geometric structure becomes explicit in the representation. However, the choice of a good dictionary in this case is far from obvious. We propose a new method for learning overcomplete dictionaries that are adapted to the joint representation of stereo images. We first formulate a sparse stereo image model where the multi-view correlation is described by local geometric transforms of dictionary elements (atoms) in two stereo views. A maximum-likelihood (ML) method for learning stereo dictionaries is then proposed, where a multi-view geometry constraint is included in the probabilistic model. The ML objective function is optimized using the expectation-maximization algorithm. We apply the learning algorithm to the case of omnidirectional images, where we learn scales of atoms in a parametric dictionary. The resulting dictionaries provide better performance in the joint representation of stereo omnidirectional images as well as improved multi-view feature matching. We finally discuss and demonstrate the benefits of dictionary learning for distributed scene representation and camera pose estimation.

We propose a new method for imaging sound speed in breast tissue from measurements obtained by ultrasound tomography (UST) scanners. Given the measurements, our algorithm finds a sparse image representation in an overcomplete dictionary that is adapted to the properties of UST images. This dictionary is learned from high resolution MRI breast scans using an unsupervised maximum likelihood dictionary learning method. The proposed dictionary-based regularization method significantly improves the quality of reconstructed breast UST images. It outperforms the wavelet-based reconstruction and the least squares minimization with lowpass constraints, on both numerical and in vivo data. Our results demonstrate that the use of the learned dictionary improves the image accuracy for up to 4 dB with the exact measurement matrix and for 3.5 dB with the estimated measurement matrix over the wavelet-based reconstruction under the same conditions.

Sets of multi-view images that capture plenoptic information from different viewpoints are typically related by geometric constraints. The proper analysis of these constraints is key to the definition of consistent compact representations of such images. We propose an algorithm for joint sparse approximation of multi-view images driven by epipolar geometry considerations. We extend greedy pursuit algorithms, such that the representation of multi-view images into linear combination of geometric atoms is able to balance approximation error and geometric consistency. We further add a rate penalty constraint that favors representations with small entropy towards efficient coding applications. Experimental results illustrate the trade-off between approximation, geometry and rate constraints in the representation of stereo omnidirectional images. In particular, we show that geometry constraints lead to a consistent description of the correlation among views, which is particularly beneficial for scene analysis or view interpolation applications. At the same time, we show that the rate constraint leads to compact representations, possibly to the detriment of geometry consistency.

This paper addresses the problem of correct recovery of multiple sparse correlated signals using distributed thresholding. We consider the scenario where multiple sensors capture the same event, but observe different signals that are correlated by local transforms of their sparse components. In this context, the signals do not necessarily have the same sparse support, but instead the support of one signal is built on local transforms of the atoms in the sparse support of another signal. We establish the sufficient condition for the correct recovery of such correlated signals using independent thresholding of the multiple signals. The condition is relevant in scenarios where low complexity processing such as thresholding is needed, for example in sensor networks. The validity of the derived recovery condition is confirmed by experimental results in noiseless and noisy scenarios.

Omnidirectional images represent a special type of images that are captured by vision sensors with a 360-degree field of view. This work targets the compression of such images by taking into account their particular geometry. We first map omnidirectional images to spherical ones and then perform sparse image decomposition over a dictionary of geometric atoms on the 2D sphere. A coder based on Matching Pursuit and adaptive quantization is finally proposed for efficient compression of omnidirectional images. The experiments demonstrate that the proposed system outperforms JPEG2000 coding of unfolded images. Since most omnidirectional sensors can be parametrized with a spherical camera model, the proposed method is generic with respect to different sensor constructions.

This paper addresses the problem of interpolating signals defined on a 2-d sphere from non-uniform samples. We present an interpolation method based on locally weighted linear and nonlinear regression, which takes into account the differences in importance of neighboring samples for signal reconstruction. We show that for optimal kernel function variance, the proposed method performs interpolation more accurately than the nearest neighbor method, especially in noisy conditions. Moreover, this method does not have memory limitations which set the upper bound on the possible interpolation points number, like in the method presented in [1].

This paper presents a distributed and occlusion-robust coding scheme for multi-view omnidirectional images, which relies on the geometry of the 3D scene. The Wyner-Ziv coder uses a multi-view correlation model that relates 3D features in different images using local geometric transforms in order to perform coset code design and the coset decoding of each feature. The meaningful image features are extracted by a sparse decomposition over a dictionary of localized geometric atoms. However, in such a decomposition, occlusions or low-correlated features appear as independent elements in the encoded stream, which can lead to erroneous reconstruction at the decoder. To ameliorate this problem, we propose to leave a controlled redundancy by sending additional syndrome bits that are computed by channel coding across the atoms of the Wyner-Ziv image. This offers resiliency against occlusions, or against inaccuracy in the view correlation model. The experimental results demonstrate the coding performance of the proposed scheme at low bit rate, where it performs close to the joint encoding strategy.

This paper addresses the problem of 3D face recognition using spherical sparse representations. We first propose a fully automated registration process that permits to align the 3D face point clouds. These point clouds are then represented as signals on the 2D sphere, in order to take benefit of the geometry information. Simultaneous sparse approximations implement a dimensionality reduction process by subspace projection. Each face is typically represented by a few spherical basis functions that are able to capture the salient facial characteristics. The dimensionality reduction step preserves the discriminant facial information and eventually permits an effective matching in the reduced space, where it can further be combined with LDA for improved recognition performance. We evaluate the 3D face recognition algorithm on the FRGC v.1.0 data set, where it outperforms classical state-of-the-art solutions based on PCA or LDA on depth face images.

This paper addresses the problem of efficient representation of scenes captured by distributed omnidirectional vision sensors. We propose a novel geometric model to describe the correlation between different views of a 3-D scene. We first approximate the camera images by sparse expansions over a dictionary of geometric atoms. Since the most important visual features are likely to be equivalently dominant in images from multiple cameras, we model the correlation between corresponding features in different views by local geometric transforms. For the particular case of omnidirectional images, we define the multiview transforms between corresponding features based on shape and epipolar geometry constraints. We apply this geometric framework in the design of a distributed coding scheme with side information, which builds an efficient representation of the scene without communication between cameras. The Wyner-Ziv encoder partitions the dictionary into cosets of dissimilar atoms with respect to shape and position in the image. The joint decoder then determines pairwise correspondences between atoms in the reference image and atoms in the cosets of the Wyner-Ziv image in order to identify the most likely atoms to decode under epipolar geometry constraints. Experiments demonstrate that the proposed method leads to reliable estimation of the geometric transforms between views. In particular, the distributed coding scheme offers similar rate-distortion performance as joint encoding at low bit rate and outperforms methods based on independent decoding of the different images.

This paper addresses the problem of compact representation of a 3D scene, captured by distributed omnidirectional cameras. As the images from the sensors are likely to be correlated in most practical scenarios, we build a distributed algorithm based on coding with side information. A reference image is processed with a wavelet transform and progressively encoded. The Wyner-Ziv images undergo a multiresolution representation, and the generated bitplanes are channel encoded with LDPC codes. The central decoder eventually reconstructs the Wyner-Ziv images given by the syndrome bits from the channel codes using the reference omnidirectional image. It also iteratively implements motion estimation on the 2-sphere in order to improve the side information. Experimental results demonstrate that distributed coding improves the rate-distortion performance for coding a set of omnidirectional images when compared to independent coding solutions. The proposed method can further be extended to the decoding of multiple Wyner-Ziv images using one single reference omnidirectional image. Hence, it achieves a reduced overall coding rate compared to disparity-based schemes. In addition, it does not require explicit knowledge of the camera parameters nor precise calibration, which is certainly interesting in camera networks.

This paper addresses the problem of distributed coding of light fields in camera networks. A novel distributed coding scheme with side information is presented, based on spherical image expansion over an over complete dictionary of geometric atoms. We propose to model the correlation between views with local geometrical transformations of corresponding features in the sparse representations of different views. We design a Wyner-Ziv encoder by partitioning the dictionary into cosets of dissimilar atoms, with respect to their shape and position on the image. The joint decoder finds pairwise correspondences between atoms in the reference image and atoms in cosets of the Wyner-Ziv image. It selects the most likely correspondence among pairs of atoms that satisfy epipolar geometry constraints. This permits to estimate local transformations between correlated images that eventually help to refine the side information provided by the reference image. Experiments demonstrate that the proposed method is capable of estimating the geometric transformations between views, and hence to reconstruct the Wyner-Ziv image.

This paper proposes a new method for the selection of sets of omnidirectional views, which contribute together to the efficient representation of a 3D scene. When the 3D surface is modelled as a function on a unit sphere, the view selection problem is mostly governed by the accuracy of the 3D surface reconstruction from non-uniformly sampled datasets. A novel method is proposed for the reconstruction of signals on the sphere from scattered data, using a generalization of the spherical Fourier transform. With that reconstruction strategy, an algorithm is then proposed to select the best subset of n views, from a predefined set of viewpoints, in order to minimize the overall reconstruction error. Starting from initial viewpoints determined by the frequency distribution of the 3D scene, the algorithm iteratively refines the selection of each of the viewpoints, in order to maximize the quality of the representation. Experiments show that the algorithm converges towards a minimal distortion, and demonstrate that the selection of omnidirectional views is consistent with the frequency characteristics of the 3D scene.

This paper presents a progressive coding scheme for 3-D objects, based on overcomplete signal expansions on the 2-D sphere. Due to increased freedom in the basis construction, redundant expansions have shown interesting approximation properties in the decomposition of signals with multidimensional singularities organized along embedded submanifolds. We propose to map simple 3-D models on 2-D spheres and then to decompose the signal over a redundant dictionary of oriented and anisotropic atoms living on the sphere. The signal expansion is computed iteratively with a matching pursuit algorithm, which greedily selects the most prominent components of the 3-D model. The decomposition therefore inherently represents a progressive stream of atoms, which is advantageously used in the design of scalable representations. An encoder is proposed that compresses the stream of atoms by adaptive coefficient quantization and entropy coding of atom indexes. Experimental results show that the novel coding strategy outperforms state-of-the-art progressive coders in terms of distortion, mostly at low bit rates. Furthermore, since the dictionary is built on structured atoms, the proposed representation simultaneously offers an increased flexibility for easy stream manipulations. We finally illustrate that advantage in the design of a view-dependent transmission scheme

Summary form only given. The paper presents a low rate progressive 3D mesh compression scheme for simple, genus-zero 3D objects. The proposed scheme is based on signal representation using redundant expansions on the 2D-sphere. First, generic input data is re-sampled as a function on the 2D-sphere, and the signal value for each point on the regular grid is obtained by performing nearest neighbor interpolation within four points from the initial 3D model. The model representation is then constructed using a matching pursuit algorithm, with an over-complete dictionary of atoms, defined on a sphere. In order to capture the particular characteristics of the 3D models efficiently, we propose a dictionary construction based on two generating functions, a Gaussian to capture low-frequency components, and a modified combination of a Gaussian and its second derivative to capture high-frequency components of the input signal. Compared to state-of-the-art encoders, our method has been shown to offer very good compression efficiency, but the performance is limited by the resampling step that maps the input model on the 2D-sphere. Matching pursuit has, however, the advantage of providing an intrinsically progressive scheme, that is also very flexible.