The fusion of multiple images from different modalities is the process of generating a single output image that combines the useful information of all input images. Ideally, the information-rich content of each input image would be preserved, and the cognitive effort required by the user to extract this information should be smaller on the fused image than the one required to examine all images. We propose MobileFuse, an edge computing method targeted at processing large amount of imagery in a bandwidth limited environment using depthwise separable Deep Neural Networks (DNNs). The proposed approach is a hybrid between generative and blending based methods. Our approach can be applied in various fields which require low latency interaction with the user or with an autonomous system. The main challenge in training DNNs for image fusion is the sparsity of data with representative ground truth. Registering images from different sensors is a major challenge in itself, and generating a ground truth from them is another massive one. For this reason, we also propose a multi-focus and multi-lighting framework to generate training dataset using unregistered images. We show that our edge network can perform faster than its state-of-the-art baseline, while improving the fusion quality.

Multimodal image registration is a challenging task. To begin with, the variation of parallax in the images makes the process intrinsically tricky. Additionally, due to phenomenology differences in modalities, the appearance of the same feature may vary significantly between the images making the registration laborious. To help mitigate these issues, we propose a two-step approach targeted at visible and infrared imagery. First, we train a generative adversarial network to learn the domain transfer function between the visible and the infrared domain, thereby mitigating the impact of the visual dissimilarity between the images. Second, we train a deep Siamese network to compute a homography in an unsupervised setting. Both elements are combined and trained sequentially. Our method is evaluated on a publicly available dataset. Our results show that the proposed method provides a reduction of more than 30% on average from the previous state-of-the-art, and outperforms several baselines and recent deep homography methods.

Anomaly detection is a key functionality in various vision systems, such as surveillance and security. In this work, we present a convolutional neural network (CNN) that supports the detection of anomaly, which has not been defined when building the model, at frame level. Our CNN, named SmithNet, is structured to simultaneously learn commonly occurring textures and their corresponding motion. Its architecture is a combination of: 1) an encoder extracting motion-texture coherence from each video frame and 2) two decoders that separately reconstruct the input as well as predict its typical motion from the estimated coherence. We also introduce an encoding block, which is specifically designed for the task of anomaly detection. The optimization is performed on only data of normal events, and the network is expected to determine the ones that are unusual, i.e., have not been seen before. According to the experiments on eight benchmark datasets of different environments with various anomalous events, the performance of our network is competitive or outperforms current state-of-the-art approaches.

Structured light-based 3D scanning presents various challenges. While robustness to indirect illumination has been the subject of recent research, little has been said about discontinuities. This paper proposes a new discontinuity-aware algorithm for estimating structured light correspondences with subpixel accuracy. The algorithm is not only robust to common structured light problems, such as indirect lighting effects, but also identifies discontinuities explicitly. This results in a significant reduction of reconstruction artifacts at objects borders, an omnipresent problem of structured light methods, especially those relying on direct decoding. Our method is faster than previously proposed robust subpixel methods, has been tested on synthetic as well as real data and shows a significant improvement on measurement at discontinuities when compared with other state-of-the-art methods.

Fisheye cameras are notoriously hard to calibrate using traditional plane-based methods. This paper proposes a new calibration method for large field of view cameras. Similarly to planar calibration, it relies on multiple images of a planar calibration grid with dense correspondences, typically obtained using structured light. By relying on the grids themselves instead of the distorted image plane, we can build a rectilinear Generic Virtual Central (GVC) camera. Instead of relying on a single GVC camera, our method proposes a selection of multiple GVC cameras which can cover any field of view and be trivially aligned to provide a very accurate generic central model. We demonstrate that this approach can directly model axial cameras, assuming the distortion center is located on the camera axis. Experimental validation is provided on both synthetic and real fisheye cameras featuring up to a 280° field of view. To our knowledge, this is one of the only practical methods to calibrate axial cameras.

This paper presents a new approach whichbuilds 360-degree cylindrical panoramic images from multiple cameras. In order to ensure a perceptually correct result, mosaicing typically requires either a planar or near-planar scene, parallax-free camera motion between source frames, or a dense sampling of the scene. When these conditions are not satisfied, various artifacts may appear. There are many algorithms to overcome these problems.We propose a panoramic setup where cameras are placed evenly around a circle. Instead of looking outward, which is the traditionalconfiguration, we propose to make the optical axes tangent to the camera circle, a "tangential" configuration.We will demonstrate that this configuration is very insensitive to depth estimation, which reduces stitching artifacts. This property is only limited by the fact that tangential cameras usually occlude each other along the circle.Beside an analysis and comparison of radial and tangential geometries, we provide an experimental setup with real panoramas obtained in realistic conditions.

This paper adresses the multiple-webcam synchronisation problem for dynamic scenes by proposing a new triangulation method which is invariant to the temporal offset of the webcams. Our method estimates the correct positions of the tracked points without knowing or directly estimating the temporal offset between the cameras. Instead, it relies on the epipolar geometry of the cameras. In the presence of a temporal delay, a tracked point does not lay on its corresponding epipolar line, thereby inducing triangulation errors. By intersecting its motion trajectory with the corresponding epipolar line, an accurate triangulation can be obtained. One major advantage of our approach is that the temporal offset can change throughout the sequence. Temporal offset variations occur frequently when using webcams. The major contribution of this work is making a temporal offset invariant triangulation solution.

An omni stereo pair of images provides depth information from stereo up to 360 degrees around a central observer. A method for synthesizing omni stereo video textures was recently introduced which was Based on blending of overlapping stereo videos that were filmed several seconds apart. While it produced loop able omni stereo videos that can be displayed up to 360 degrees around a viewer, ghosting was visible within blended overlaps. This paper presents a stereo stitching method to render these overlaps without any ghosting. The stitching method uses a graph-cut minimization. We show results for scenes with differents types of motion, such as water flows, leaves in the wind and moving people.

We introduce in this paper a camera setup for stereo immersive (omnistereo) capture. An omnistereo pair of images gives stereo information up to 360 degrees around a central observer. Previous methods to produce omnistereo images assume a static scene in order to stitch together multiple images captured by a stereo camera rotating on a fixed tripod. Our omnipolar camera setup uses a minimum of 3 cameras with fisheye lenses. The multiple epipoles are used as locations to stitch the images together and produce omnistereo images with no horizontal misalignments due to parallax. We show results of using 3 cameras to capture an unconstrained dynamic scene while the camera is travelling. The produced omnistereo videos are formatted to be displayed on a cylindrical screen or dome.

We present a scanning method that recovers dense sub pixel camera-projector correspondence without requiring any photometric calibration nor preliminary knowledge of their relative geometry. Sub pixel accuracy is achieved by considering several zero-crossings defined by the difference between pairs of unstructured patterns. We use gray-level band-pass white noise patterns that increase robustness to indirect lighting and scene discontinuities. Simulated and experimental results show that our method recovers scene geometry with high sub pixel precision, and that it can handle many challenges of active reconstruction systems. We compare our results to state of the art methods such as micro phase shifting and modulated phase shifting.

Most methods for synthesizing panoramas assume that the scene is static. A few methods have been proposed for synthesizing stereo or motion panoramas, but there has been little attempt to synthesize panoramas that have both stereo and motion. One faces several challenges in synthesizing stereo motion panoramas, for example, to ensure temporal synchronization between left and right views in each frame, to avoid spatial distortion of moving objects, and to continuously loop the video in time. We have recently developed a stereo motion panorama method that tries to address some of these challenges. The method blends space-time regions of a video XYT volume, such that the blending regions are distinct and translate over time. This article presents a perception experiment that evaluates certain aspects of the method, namely how well observers can detect such blending regions. We measure detection time thresholds for different blending widths and for different scenes, and for monoscopic versus stereoscopic videos. Our results suggest that blending may be more effective in image regions that do not contain coherent moving objects that can be tracked over time. For example, we found moving water and partly transparent smoke were more effectively blended than swaying branches. We also found that performance in the task was roughly the same for mono versus stereo videos.

A panoramic stereo (or omnistereo) pair of images provides depth information from stereo up to 360 degrees around a central observer. Because omnistereo lenses or mirrors do not yet exist, synthesizing omnistereo images requires multiple stereo camera positions and baseline orientations. Recent omnistereo methods stitch together many small field of view images called slits which are captured by one or two cameras following a circular motion. However, these methods produce omnistereo images for static scenes only. The situation is much more challenging for dynamic scenes since stitching needs to occur over both space and time and should synchronize the motion between left and right views as much as possible. This paper presents the first ever method for synthesizing panoramic stereo video textures. The method uses full frames rather than slits and uses blending across seams rather than smoothing or matching based on graph cuts. The method produces loopable panoramic stereo videos that can be displayed up to 360 degrees around a viewer.

Reconstruction from structured light can be greatly affected by interreflections between surfaces in the scene. This paper introduces band-pass white noise patterns designed specifically to reduce interreflections, and still be robust to standard challenges in scanning systems such as scene depth discontinuities, defocus and low camera-projector pixel ratio. While this approach uses unstructured light patterns that increase the number of required projected images, it is up to our knowledge the first method that is able to recover scene disparities in the presence of both scene discontinuities and interreflections. Furthermore, the method does not require calibration (geometric nor photometric) or post-processing such as dynamic programming or phase unwrapping. We show results for a challenging scene and compare them to correspondences obtained with the well-known Gray code and Phase-shift methods.

An omnistereoscopic image is a pair of panoramic images that enables stereoscopic depth perception all around an observer. An omnistereo projection on a cylindrical display does not require tracking of the observer's viewing direction. However, such a display introduces stereo distortions. In this article, we investigate two projection models for rendering 3D scenes in omnistereo. The first is designed to give zero disparity errors at the center of the visual field. The second is the well-known slit-camera model. For both models, disparity errors are shown to increase gradually in the periphery, as visual stereo acuity decreases. We use available data on human stereoscopic acuity limits to argue that depth distortions caused by these models are so small that they cannot be perceived.

We present algorithms for plane-based calibration of general radially distorted cameras. By this, we understand cameras that have a distortion center and an optical axis such that the projection rays of pixels lying on a circle centered on the distortion center form a right viewing cone centered on the optical axis. The camera is said to have a single viewpoint (SVP) if all such viewing cones have the same apex (the optical center); otherwise, we speak of NSVP cases. This model encompasses the classical radial distortion model [5], fisheyes, and most central or noncentral catadioptric cameras. Calibration consists in the estimation of the distortion center, the opening angles of all viewing cones, and their optical centers. We present two approaches of computing a full calibration from dense correspondences of a single or multiple planes with known euclidean structure. The first one is based on a geometric constraint linking viewing cones and their intersections with the calibration plane (conic sections). The second approach is a homography-based method. Experiments using simulated and a broad variety of real cameras show great stability. Furthermore, we provide a comparison with Hartley-Kang's algorithm [12], which, however, cannot handle such a broad variety of camera configurations, showing similar performance.

Matrix factorization is a key component for solving several computer vision problems. It is particularly challenging in the presence of missing or erroneous data, which often arise in structure-from-motion. We propose batch algorithms for matrix factorization. They are based on closure and basis constraints, that are used either on the cameras or the structure, leading to four possible algorithms. The constraints are robustly computed from complete measurement sub-matrices with e.g. random data sampling. The cameras and 3D structure are then recovered through linear least squares. Prior information about the scene such as identical camera positions or orientations, smooth camera trajectory, known 3D points and coplanarity of some 3D points can be directly incorporated. We demonstrate our algorithms on challenging image sequences with tracking error and more than 95% missing data.

We present an algorithm for plane-based self-calibration of cameras with radially symmetric distortions given a set of sparse feature matches in at least two views. The projection function of such cameras can be seen as a projection with a pinhole camera, followed by a non-parametric displacement of the image points in the direction of the distortion center. The displacement is a function of the points' distance to the center. Thus, the generated distortion is radially symmetric. Regular cameras, fish-eyes as well as the most popular central catadioptric devices can be described by such a model. Our approach recovers a distortion function consistent with all the views, or estimates one for each view if they are taken by different cameras. We consider a least squares algebraic solution for computing the homography between two views that is valid for rectified (undistorted) point correspondences. We observe that the terms of the function are bilinear in the unknowns of the homography and the distortion coefficient associated to each point. Our contribution is to approximate this non-convex problem by a convex one. To do so, we replace the bilinear terms by a set of new variables and obtain a linear least squares problem. We show that like the distortion coefficients, these variables are subject to monotonicity constraints. Thus, the approximate problem is a convex quadratic program. We show that solving it is sufficient for accurately estimating the distortion parameters. We validate our approach on simulated data as well as on fish-eye and catadioptric cameras. We also compare our solution to three state-of-the-art algorithms and show similar performance.

This paper presents a novel algorithm that improves the localization of disparity discontinuities of disparity maps obtained by multi-baseline stereo. Rather than associating a disparity label to every pixel of a disparity map, it associates a position to every disparity discontinuity. This formulation allows us to find an approximate solution to a 2D labeling problem with robust smoothing term by minimizing multiple 1D problems, thus making possible the use of dynamic programming. Dynamic programming allows the efficient computation of the visibility of most of the cameras during the minimization. Whilst the proposed minimization strategy is particularly suitable for stereo with occlusion, it may be used with other labeling problems.

We present a 3D reconstruction technique based on the maximum-flow formulation. Starting with a set of calibrated images, we globally search for the most probable 3D model given the photoconsistency and the spatial continuity constraints. This search is done radially from the center of the reconstruction volume; therefore imposing a radial topology. The fact that cameras are arbitrarily positioned around the scene presents challenges for managing occlusion, especially when applying global smoothing. We solve this problem by proposing an iterative occlusion management mechanism, and a new way of looking at surface smoothing and discontinuities that takes photoconsistency into account. Experiments show that our method is relatively fast and robust when dealing with simple objects, even in noisy conditions. 1

This paper presents a new model to overcome the occlusion problems coming from wide baseline multiple camera stereo. Rather than explicitly modeling occlusions in the matching cost function, it detects occlusions in the depth map obtained from regular efficient stereo matching algorithms. Occlusions are detected as inconsistencies of the depth map by computing the visibility of the map as it is reprojected into each camera. Our approach has the particularity of not discriminating between occluders and occludees. The matching cost function is modified according to the detected occlusions by removing the offending cameras from the computation of the matching cost. The algorithm gradually modifies the matching cost function according to the history of inconsistencies in the depth map, until convergence. While two graph-theoretic stereo algorithms are used in our experiments, our framework is general enough to be applied to many others. The validity of our framework is demonstrated using real imagery with different baselines.

Many applications in computer vision use Principal Components Analysis (PCA), for example, in camera calibration, stereo, localization and motion estimation. We present a new and fast PCA-based method to analyze optical snow. Optical snow is a complex form of visual motion that occurs when an observer moves through a highly cluttered 3D scene. For this category of motion field, no spatial or depth coherence can be assumed. Previous methods for measuring optical snow have used a wedge filter in a spatiotemporal frequency domain. The PCA method is also based on the spatiotemporal frequency domain analysis, but examines a different geometry property of the spectrum. We compare the results of the PCA method to the previous methods using both real and synthetic sequences. 1

The paper presents a new method to correct overlapping textures of a single 3D model where each texture was obtained under possibly different lighting conditions and color response of the camera. This situation arises frequently when a single object is digitized using multiple 3D scanners. Our goal is to remove any visible seam and improve color consistency between the textures in order to merge them afterwards. To achieve this, we propose an efficient algorithm to compute the "ratio lighting" of two textures, derive a common lighting from it, and use it to "relight" each texture. We illustrate our method by correcting textures of human faces acquired with several structured-light 3D scanners. Experimental results are realistic and demonstrate how this method can reduce the need to calibrate colors or explicitly solve for the illumination.

We propose an efficient, non-iterative method for estimating optical flow. We develop a probabilistic framework that is appropriate for describing the inherent uncertainty in the brightness constraint due to errors in image derivative computation. We separate the flow into two 1D representations and pose the problem of flow estimation as one of solving for the most probable configuration of 1D labels in an Markov random fields (MRF) with linear clique potentials. The global optimum for this problem can be efficiently solved for using the maximum flow computation in a graph. We develop this formulation and describe how the use of the probabilistic framework, the parametrisation and MRF formulation together enables one to capture the desirable properties for flow estimation, especially preserving motion discontinuities. We demonstrate the performance of our algorithm and compare our results with that of other algorithms described by Barron et. al. (1994).

This paper describes a new algorithm for solving the N-camera stereo correspondence problem by transforming it into a maximum-flow problem. Once solved, the minimum-cut associated to the maximum-flow yields a disparity surface for the whole image at once. This global approach to stereo analysis provides a more accurate and coherent depth map than the traditional line-by-line stereo. Moreover, the optimality of the depth surface is guaranteed and can be shown to be a generalization of the dynamic programming approach that is widely used in standard stereo. Results show improved depth estimation as well as better handling of depth discontinuities. While the worst case running time is O(n/sup 2/d/sup 2/log(nd)), the observed average running time is O(n/sup 1.2/ d/sup 1.3/) for an image size of n pixels and depth resolution d.

We propose anew rectification method for aligning epipolar lines of a pair of stereo images taken under any camera geometry. It effectively remaps both images onto the surface of a cylinder instead of a plane, which is used in common rectification methods. For a large set of camera motions, remapping to a plane has the drawback of creating rectified images that are potentially infinitely large and presents a loss of pixel information along epipolar lines. In contrast, cylindrical rectification guarantees that the rectified images are bounded for all possible camera motions and minimizes the loss of pixel information along epipolar line. The processes (e.g., stereo matching, etc.) subsequently applied to the rectified images are thus more accurate and general since they can accommodate any camera geometry.

We propose a new paradigm, motion without structure, for determining the ego-motion between two frames. It is best suited for cases where reliable feature point correspondence is difficult, or for cases where the expected camera motion is large. The problem is posed as a five-dimensional search over the space of possible motions during which the structural information present in the two views is neither implicitly or explicitly used or estimated. To accomplish this search, a cost function is devised that measures the relative likelihood of each hypothesized motion. This cost function is invariant to the structure present in the scene. An analysis of the global scene statistics present in an image, together with the geometry of epipolar misalignment, suggests a measure based on the sum of squared differences between pixels in the first image and their corresponding epipolar line segments in the second image. The measure relies on a simple statistical characteristic of neighboring image intensity levels. Specifically, that the variance of intensity differences between two arbitrary points in an image is a monotonically increasing symmetrical function of the distance between the two points. This assumption is almost always true, though the size of the neighborhood over which the monotonic dependency holds varies from image to image. This range determines the maximum permissible motion between two frames, which can be quite large. Experiments with both outdoor scenes and an indoor calibrated sequence achieve very good accuracy (less then 1 pixel image displacement error) and robustness to noise.

The constant image brightness (CIB) assumption assumes that the intensities of corresponding points in two images are equal. This assumption is central to much of computer vision. However, surprisingly little work has been performed to support this assumption, despite the fact the many of algorithms are very sensitive to deviations from CIB. An examination of the images contained in the SRI JISCT stereo database revealed that the constant image brightness assumption is indeed often false. Moreover, the simple additive/multiplicative models of the form I/sub L/=/spl beta/I/sub R/+/spl alpha/ do not adequately represent the observed deviations. A comprehensive physical model of the observed deviations is difficult to develop. However, many potential sources of deviations can be represented by a nonlinear monotonically increasing relationship between intensities. Under these conditions, we believe that an expansion/contraction matching of the intensity histograms represents the best method to both measure the degree of validity of the CIB assumption and correct for it. Dynamic histogram warping (DHW) is closely related to histogram specification. It is shown that histogram specification introduces artifacts that do not occur with dynamic histogram warping. Experimental results show that image histograms are closely matched after DHW, especially when both histograms are modified simultaneously. DHW is also capable of removing simple constant additive and multiplicative biases without derivative operations, thereby avoiding amplification of high frequency noise. DHW can improve the estimates from stereo and optical flow estimators.