In structure from motion or similarly in monocular SLAM problems, spatio-temporal image variations, motion and scene geometry are intimately related and in the absence of such a constraint, unsupervised deep learning methods often tend to state the problem under multiple constraints. We readdress the problem of 3D interpretation estimation as an unsupervised deep learning process where depth and camera motion are learned to satisfy the 3D brightness constraint for rigid objects. We introduce for the first time a new learning paradigm where the spatio-temporal variations of image sequences are coupled to 3D interpretation to minimize the loss without need to add more ad-hoc constraints that are not related to the 3D interpretation. Experimental results show that our method competes and sometimes outperforms the state-of-the-art methods.

Projecting an image onto an arbitrary non-flat screen surface leads to undesired geometric distortions in the image's projection. Geometric correction pre-distorts the image being projected such that the image's projection appears geometrically correct. In this work, we propose a novel saliency-guided projection geometric correction (SPGC) method that leverages calibration parameters along with 3D surface geometry captured by the projector-camera system to compensate for the geometric distortions created by non-flat screen surfaces. The proposed SPGC method incorporates a novel sampling scheme that selects a small set of surface points for geometric correction estimation based on local surface saliency, which greatly reduces the computational complexity of geometric correction estimation process. Experimental results using a test non-flat screen surface with abrupt edges and curve showed that the proposed SPGC approach achieved superior distortion compensation performance both quantitatively and qualitatively when compared to an unguided projection geometric correction method, while requiring just 3% of the samples used by a conventional densely-sampled projection geometric correction method.

Utilization of an acoustic camera for range measurements is a key advantage for 3-D shape recovery of underwater targets by opti-acoustic stereo imaging, where the associated epipolar geometry of optical and acoustic image correspondences can be described in terms of conic sections. In this paper, we propose methods for system calibration and 3-D scene reconstruction by maximum likelihood estimation from noisy image measurements. The recursive 3-D reconstruction method utilized as initial condition a closed-form solution that integrates the advantages of two other closed-form solutions, referred to as the range and azimuth solutions. Synthetic data tests are given to provide insight into the merits of the new target imaging and 3-D reconstruction paradigm, while experiments with real data confirm the findings based on computer simulations, and demonstrate the merits of this novel 3-D reconstruction paradigm.

Target-based positioning and 3-D target reconstruction are critical capabilities in deploying submersible platforms for a range of underwater applications, e.g., search and inspection missions. While optical cameras provide high-resolution and target details, they are constrained by limited visibility range. In highly turbid waters, target at up to distances of 10 s of meters can be recorded by high-frequency (MHz) 2-D sonar imaging systems that have become introduced to the commercial market in years. Because of lower resolution and SNR level and inferior target details compared to optical camera in favorable visibility conditions, the integration of both sensing modalities can enable operation in a wider range of conditions with generally better performance compared to deploying either system alone. In this paper, estimate of the 3-D motion of the integrated system and the 3-D reconstruction of scene features are addressed. We do not require establishing matches between optical and sonar features, referred to as opti-acoustic correspondences, but rather matches in either the sonar or optical motion sequences. In addition to improving the motion estimation accuracy, advantages of the system comprise overcoming certain inherent ambiguities of monocular vision, e.g., the scale-factor ambiguity, and dual interpretation of planar scenes. We discuss how the proposed solution provides an effective strategy to address the rather complex opti-acoustic stereo matching problem. Experiment with real data demonstrate our technical contribution.

Utilization of an acoustic camera for range measurements is a key advantage for 3-D shape recovery of underwater targets by opti-acoustic stereo imaging, where the associated epipolar geometry of optical and acoustic image correspondences can be described in terms of conic sections. In this paper, we propose methods for system calibration and 3-D scene reconstruction by maximum likelihood estimation from noisy image measurements. The recursive 3-D reconstruction method utilized as initial condition a closed-form solution that integrates the advantages of so-called range and azimuth solutions. Synthetic data tests are given to provide insight into the merits of the new target imaging and 3-D reconstruction paradigm, while experiments with real data confirm the findings based on computer simulations, and demonstrate the merits of this novel 3-D reconstruction paradigm.

This study investigates a variational, active curve evolution method for dense three-dimentional (3D) segmentation and interpretation of optical flow in an image sequence of a scene containing moving rigid objects viewed by a possibly moving camera. This method jointly performs 3D motion segmentation, 3D interpretation (recovery of 3D structure and motion), and optical flow estimation. The objective functional contains two data terms for each segmentation region, one based on the motion-only equation which relates the essential parameters of 3D rigid body motion to optical flow, and the other on the Horn and Schunck optical flow constraint. It also contains two regularization terms for each region, one for optical flow, the other for the region boundary. The necessary conditions for a minimum of the functional result in concurrent 3D-motion segmentation, by active curve evolution via level sets, and linear estimation of each region essential parameters and optical flow. Subsequently, the screw of 3D motion and regularized relative depth are recovered analytically for each region from the estimated essential parameters and optical flow. Examples are provided which verify the method and its implementation.

The purpose of this study is to investigate a variational method for joint multiregion three-dimensional (3-D) motion segmentation and 3-D interpretation of temporal sequences of monocular images. Interpretation consists of dense recovery of 3-D structure and motion from the image sequence spatiotemporal variations due to short-range image motion. The method is direct insomuch as it does not require prior computation of image motion. It allows movement of both viewing system and multiple independently moving objects. The problem is formulated following a variational statement with a functional containing three terms. One term measures the conformity of the interpretation within each region of 3-D motion segmentation to the image sequence spatiotemporal variations. The second term is of regularization of depth. The assumption that environmental objects are rigid accounts automatically for the regularity of 3-D motion within each region of segmentation. The third and last term is for the regularity of segmentation boundaries. Minimization of the functional follows the corresponding Euler-Lagrange equations. This results in iterated concurrent computation of 3-D motion segmentation by curve evolution, depth by gradient descent, and 3-D motion by least squares within each region of segmentation. Curve evolution is implemented via level sets for topology independence and numerical stability. This algorithm and its implementation are verified on synthetic and real image sequences. Viewers presented with anaglyphs of stereoscopic images constructed from the algorithm's output reported a strong perception of depth.

The aim of this study is to introduce a novel method for the simultaneous motion segmentation and dense 3D interpretation of temporal sequences of monocular images. The problem is to recover simultaneously 3D structure, 3D motion, and a motion-based segmentation from the image sequence spatio-temporal variations. Motion in space is considered relative to the viewing system so that both the viewing system and environmental objects are allowed to move. The problem is stated as a 3D motion segmentation problem with simultaneous depth estimation within the regions of segmentation. The Euler-Lagrange equations of minimization of the objective functional lead to curve evolution PDE implemented via level sets.