Measuring the 3D shape of semi-transparent surfaces with projector-camera 3D scanners is a difficult task because these surfaces weakly reflect light in a diffuse manner, and transmit a large part of the incident light. The task is even harder in the presence of participating background surfaces. The two methods proposed in this paper use sinusoidal patterns, each with a frequency chosen in the frequency range allowed by the projection optics of the projector-camera system. They differ in the way in which the camera-projector correspondence map is established, as well as in the number of patterns and the processing time required. The first method utilizes the discrete Fourier transform, performed on the intensity signal measured at a camera pixel, to inventory projector columns illuminating directly and indirectly the scene point imaged by that pixel. The second method goes beyond discrete Fourier transform and achieves the same goal by fitting a proposed analytical model to the measured intensity signal. Once the one (camera pixel) to many (projector columns) correspondence is established, a surface continuity constraint is applied to extract the one to one correspondence map linked to the semi-transparent surface. This map is used to determine the 3D point cloud of the surface by triangulation. Experimental results demonstrate the performance (accuracy, reliability) achieved by the proposed methods.

The fusion of multiple images from different modalities is the process of generating a single output image that combines the useful information of all input images. Ideally, the information-rich content of each input image would be preserved, and the cognitive effort required by the user to extract this information should be smaller on the fused image than the one required to examine all images. We propose MobileFuse, an edge computing method targeted at processing large amount of imagery in a bandwidth limited environment using depthwise separable Deep Neural Networks (DNNs). The proposed approach is a hybrid between generative and blending based methods. Our approach can be applied in various fields which require low latency interaction with the user or with an autonomous system. The main challenge in training DNNs for image fusion is the sparsity of data with representative ground truth. Registering images from different sensors is a major challenge in itself, and generating a ground truth from them is another massive one. For this reason, we also propose a multi-focus and multi-lighting framework to generate training dataset using unregistered images. We show that our edge network can perform faster than its state-of-the-art baseline, while improving the fusion quality.

Multimodal image registration is a challenging task. To begin with, the variation of parallax in the images makes the process intrinsically tricky. Additionally, due to phenomenology differences in modalities, the appearance of the same feature may vary significantly between the images making the registration laborious. To help mitigate these issues, we propose a two-step approach targeted at visible and infrared imagery. First, we train a generative adversarial network to learn the domain transfer function between the visible and the infrared domain, thereby mitigating the impact of the visual dissimilarity between the images. Second, we train a deep Siamese network to compute a homography in an unsupervised setting. Both elements are combined and trained sequentially. Our method is evaluated on a publicly available dataset. Our results show that the proposed method provides a reduction of more than 30% on average from the previous state-of-the-art, and outperforms several baselines and recent deep homography methods.

Current airborne camera systems (ACS) used for surveillance and reconnaissance are generally equipped with cameras operating in different wavelengths. The registration and fusion of the images generated by these cameras enable a multimodal detection capability. In this paper, we propose a method to perform the geometric registration of infrared and visible imagery collected with an electro-optical and infrared (EO-IR) ACS. The registration process is a two-step approach. First, an initial transformation is computed using the expected orientation of the scene with respect to the camera and the camera’s intrinsic and extrinsic parameters. Then, the image alignment is refined using a technique based on mutual information which adapts the registration to the scene’s geometry and copes with video streams that are not perfectly synchronized. The image registration technique is implemented on an edge computing device and currently runs at 8 Hz. The intent is to use the resulting system to augment the capability of ACS that do no have built-in image registration and fusion capabilities. We present results obtained by applying the registration approach on real airborne imagery. We also evaluate the image registration error using physical markers installed on a building.

Current mission planning simulations are typically carefully designed around a tradeoff between tight integration with accurate geospatial data or meeting real-time demands. Our proposed system is designed to achieve both. Deriving from work on cloud geospatial stores and mission planning systems, we are developing a platform for consuming new data, building it into models accessible from a real-time simulation server, and constructing machine learning systems over this data. This combination of large datasets with continuous pipelines feeding a real-time simulation forces us to bring together innovations on both the cloud infrastructure and user interface level.

We present the Frequency Shift Method, a new structured light technique allowing 3D shape acquisition in the presence of strong interreflections. The intensity signal of each camera pixel is represented by one or several peaks in the Fourier domain. If there is no interreflection, only a single peak appears, otherwise several peaks are present. Each peak represents a projector pixel participating in the illumination of the surface point imaged at the camera pixel. In the baseline version of the proposed approach, the number of patterns required is proportional to the number of projector pixels. We also propose a modification that significantly reduces the required number of patterns by subdividing and encoding the projector into multiple virtual low-resolution projectors. A method based on dynamic programming is used to separate direct and indirect illuminations. Our experimental results illustrate the effectiveness of our method compared to existing ones.

Phase-shift profilometry (PSP) is a technique for acquiring non-contact three-dimensional measurements using a projector and a camera. The technique can be used to measure 3D surfaces with high accuracy. However, it can be sensitive to motion because phase images are sequentially captured. We propose a new technique that uses residual phase errors to minimize the influence of motion on the recovered phases (and 3D points). The method minimizes residual phase error with respect to apparent phase offset variations caused by motion. Furthermore, a total variation scheme is used to minimize spatial variations of apparent phase offset variations. We demonstrate the method with different moving objects, and we also study the effect of the magnitude of the motion on the accuracy of the recovered phases. Visual artefacts as well as accuracy loss associated with motion are found to be significantly decreased.

This paper proposes a hybrid software-hardware high-resolution projection system for 3D imaging based on fringe projection. The proposed solution combines the advantages of a digital projection with those of an analogue one. It is programmable and allows a high projection rate by avoiding mechanical displacements in the projection system. Moreover, it does not suffer from the limitations of digital systems such as the presence of inter-pixel gaps and limited resolution. The proposed projection system is relatively inexpensive to build since it is composed of a simple arrangement off-the-shelf components. The system is a combination of a low-resolution digital device such as a DMD, LCoS or LCD, some optical components and software to generate the fringe patterns. A prototype of a 3D scanner based on the proposed projection system is used to asses the fitness of the proposed technology.

This paper proposes a novel and efficient representation of the point-spread function (PSF) well suited for structured light systems used in digital profilometry. For this particular application the PSF is spatially variant over the working volume and the current parametric as well as non-parametric representations are insufficient. We validate our method using a standard data set and imagery acquired using an off-the-shelf multimedia projector and an industrial camera.

This paper presents a novel algorithm that improves the localization of disparity discontinuities of disparity maps obtained by multi-baseline stereo. Rather than associating a disparity label to every pixel of a disparity map, it associates a position to every disparity discontinuity. This formulation allows us to find an approximate solution to a 2D labeling problem with robust smoothing term by minimizing multiple 1D problems, thus making possible the use of dynamic programming. Dynamic programming allows the efficient computation of the visibility of most of the cameras during the minimization. Whilst the proposed minimization strategy is particularly suitable for stereo with occlusion, it may be used with other labeling problems.

This paper presents a new model to overcome the occlusion problems coming from wide baseline multiple camera stereo. Rather than explicitly modeling occlusions in the matching cost function, it detects occlusions in the depth map obtained from regular efficient stereo matching algorithms. Occlusions are detected as inconsistencies of the depth map by computing the visibility of the map as it is reprojected into each camera. Our approach has the particularity of not discriminating between occluders and occludees. The matching cost function is modified according to the detected occlusions by removing the offending cameras from the computation of the matching cost. The algorithm gradually modifies the matching cost function according to the history of inconsistencies in the depth map, until convergence. While two graph-theoretic stereo algorithms are used in our experiments, our framework is general enough to be applied to many others. The validity of our framework is demonstrated using real imagery with different baselines.