Tasks such as autonomous navigation, 3D reconstruction, and object recognition near the water surfaces are crucial in marine robotics applications. However, challenges arise due to dynamic disturbances, e.g., light reflections and refraction from the random air-water interface, irregular liquid flow, and similar factors, which can lead to potential failures in perception and navigation systems. Traditional computer vision algorithms struggle to differentiate between real and virtual image regions, significantly complicating tasks. A virtual image region is an apparent representation formed by the redirection of light rays, typically through reflection or refraction, creating the illusion of an object’s presence without its actual physical location. This work proposes a novel approach for segmentation on real and virtual image regions, exploiting synthetic images combined with domain-invariant information, a Motion Entropy Kernel, and Epipolar Geometric Consistency. Our segmentation network does not need to be re-trained if the domain changes. We show this by deploying the same segmentation network in two different domains: simulation and the real world. By creating realistic synthetic images that mimic the complexities of the water surface, we provide fine-grained training data for our network (MARVIS) to discern between real and virtual images effectively. By motion & geometry-aware design choices and through comprehensive experimental analysis, we achieve state-of-the-art real-virtual image segmentation performance in unseen real world domain, achieving an IoU over 78% and a F1-Score over 86% while ensuring a small computational footprint. MARVIS offers over 43 FPS (8 FPS) inference rates on a single GPU (CPU core). Our code and dataset are available here https://github.com/jiayi-wu-umd/MARVIS.

In scattering media, as in underwater or haze and fog in atmosphere, image contrast deteriorates significantly due to backscatter. This adversely affects the performance of many computer vision techniques developed for clear open-air conditions, including stereo matching, when applied to images acquired in these environments. Since the strength of the scattering depends on the distance to the scene points, the scattering field embodies range information that can be exploited for 3-D reconstruction. In this paper, we present an integrated solution for 3-D structure from stereovision that incorporates the visual cues from both disparity and scattering. The method applies to images of scenes illuminated by artificial sources and natural lighting, and performance improves with discrepancy between the backscatter fields in the two views. Neither source calibration nor knowledge of medium optical properties is required. Instead, backscatter fields at infinity, i.e., stereo images taken with no target in the field of view, are directly employed in the estimation process. Results from experiments with synthetic and real data demonstrate the key advantages of our method.

Visual odometry involves the computation of 3-D motion and (or) trajectory by tracking features in the video or image sequences recorded by the camera(s) on some autonomous terrestrial, aerial, and marine robotics platform. For exploration, mapping, inspection, and surveillance operations within turbid waters, high-frequency 2-D forward-scan sonar systems offer a significant advantage over cameras by providing both imagery with target details and attractive tradeoff in range, resolution, and data rate. Operating these at grazing incidence gives larger scene coverage and improved image quality due to the dominance of diffuse backscattered reflectance but induces cast shadows that are typically more distinct than brightness patterns due to the direct reflectance of casting objects. For the computation of 3-D motion by automatic video processing, the estimation accuracy and robustness can be enhanced by integrating the visual cues from shadow dynamics with the image flow of stationary 3-D objects, both induced by sonar motion. In this paper, we present the mathematical models of image flow for 3-D objects and their cast shadows, utilize them in devising various 3-D sonar motion estimation solutions, and study their robustness. We present results of experiments with both synthetic and real data in order to assess the accuracy and performance of these methods.

Underwater, natural illumination typically varies strongly temporally and spatially. The reason is that waves on the water surface refract light into the water in a spatiotemporally varying manner. The resulting underwater illumination field is known as underwater caustics or flicker. In past studies, flicker has often been considered to be an undesired effect, which degrades the quality of images. In contrast, in this work, we show that flicker can actually be useful for vision in the underwater domain. Specifically, it solves very simply, accurately, and densely the stereo correspondence problem, irrespective of the object's texture. The temporal radiance variations due to flicker are unique to each object point, thus disambiguating the correspondence, with very simple calculations. This process is further enhanced by compounding the spatial variability in the flicker field. The method is demonstrated by underwater in-situ experiments.

Consider photography in scattering media. One goal is to enhance the images and compensate for scattering effects. A second goal is to estimate a distance map of the scene. A prior method exists to achieve these goals. It is based on acquiring two images from a fixed position, using a single camera mounted with a polarizer at different settings. However, the shortcomings of this polarization-based method comprise having to acquire these images sequentially, reduced light level, and inapplicability at low backscatter degree of polarization. In this paper, a new technique is described to alleviate these issues by integrating polarization and stereo cues. More precisely, the earlier single-camera method is extended to a pair of cameras displaced by a finite baseline. Each camera utilizes polarizers at different settings. Stereo disparity and polarization analysis are fused to construct de-scattered left and right views. The binocular stereo cues provide additional geometric constraints for distance computation. Moreover, the proposed technique acquires the two raw images simultaneously. Thus it can be applied to dynamic scenes. Underwater experiments are presented.

Utilization of an acoustic camera for range measurements is a key advantage for 3-D shape recovery of underwater targets by opti-acoustic stereo imaging, where the associated epipolar geometry of optical and acoustic image correspondences can be described in terms of conic sections. In this paper, we propose methods for system calibration and 3-D scene reconstruction by maximum likelihood estimation from noisy image measurements. The recursive 3-D reconstruction method utilized as initial condition a closed-form solution that integrates the advantages of two other closed-form solutions, referred to as the range and azimuth solutions. Synthetic data tests are given to provide insight into the merits of the new target imaging and 3-D reconstruction paradigm, while experiments with real data confirm the findings based on computer simulations, and demonstrate the merits of this novel 3-D reconstruction paradigm.

Target-based positioning and 3-D target reconstruction are critical capabilities in deploying submersible platforms for a range of underwater applications, e.g., search and inspection missions. While optical cameras provide high-resolution and target details, they are constrained by limited visibility range. In highly turbid waters, target at up to distances of 10 s of meters can be recorded by high-frequency (MHz) 2-D sonar imaging systems that have become introduced to the commercial market in years. Because of lower resolution and SNR level and inferior target details compared to optical camera in favorable visibility conditions, the integration of both sensing modalities can enable operation in a wider range of conditions with generally better performance compared to deploying either system alone. In this paper, estimate of the 3-D motion of the integrated system and the 3-D reconstruction of scene features are addressed. We do not require establishing matches between optical and sonar features, referred to as opti-acoustic correspondences, but rather matches in either the sonar or optical motion sequences. In addition to improving the motion estimation accuracy, advantages of the system comprise overcoming certain inherent ambiguities of monocular vision, e.g., the scale-factor ambiguity, and dual interpretation of planar scenes. We discuss how the proposed solution provides an effective strategy to address the rather complex opti-acoustic stereo matching problem. Experiment with real data demonstrate our technical contribution.

Utilization of an acoustic camera for range measurements is a key advantage for 3-D shape recovery of underwater targets by opti-acoustic stereo imaging, where the associated epipolar geometry of optical and acoustic image correspondences can be described in terms of conic sections. In this paper, we propose methods for system calibration and 3-D scene reconstruction by maximum likelihood estimation from noisy image measurements. The recursive 3-D reconstruction method utilized as initial condition a closed-form solution that integrates the advantages of so-called range and azimuth solutions. Synthetic data tests are given to provide insight into the merits of the new target imaging and 3-D reconstruction paradigm, while experiments with real data confirm the findings based on computer simulations, and demonstrate the merits of this novel 3-D reconstruction paradigm.

Many applications in mobile and underwater robotics employ 3D vision techniques for navigation and mapping. These techniques usually involve the extraction and 3D reconstruction of scene interest points. Nevertheless, in large environments the huge volume of acquired information could pose serious problems to real-time data processing. Moreover, In order to minimize the drift, these techniques use data association to close trajectory loops, decreasing the uncertainties in estimating the position of the robot and increasing the precision of the resulting 3D models. When faced to large amounts of features, the efficiency of data association decreases drastically, affecting the global performance. This paper proposes a framework that highly reduces the number of extracted features with minimum impact on the precision of the 3D scene model. This is achieved by minimizing the representation redundancy by analyzing the geometry of the environment and extracting only those features that are both photometrically and geometrically significant

Optical and acoustic cameras are suitable imaging systems to inspect underwater structures, both in regular maintenance and security operations. Despite high resolution, optical systems have limited visibility range when deployed in turbid waters. In contrast, the new generation of high-frequency (MHz) acoustic cameras can provide images with enhanced target details in highly turbid waters, though their range is reduced by one to two orders of magnitude compared to traditional low-/midfrequency (10s-100s KHz) sonar systems. It is conceivable that an effective inspection strategy is the deployment of both optical and acoustic cameras on a submersible platform, to enable target imaging in a range of turbidity conditions. Under this scenario and where visibility allows, registration of the images from both cameras arranged in binocular stereo configuration provides valuable scene information that cannot be readily recovered from each sensor alone. We explore and derive the constraint equations for the epipolar geometry and stereo triangulation in utilizing these two sensing modalities with different projection models. Theoretical results supported by computer simulations show that an opti-acoustic stereo imaging system outperforms a traditional binocular vision with optical cameras, particularly for increasing target distance and (or) turbidity.

This paper presents a novel approach for combining a set of registered images into a composite mosaic with no visible seams and minimal texture distortion. To promote execution speed in building large area mosaics, the mosaic space is divided into disjoint regions of image intersection based on a geometric criterion. Pair-wise image blending is performed independently in each region by means of watershed segmentation and graph cut optimization. A contribution of this work – use of watershed segmentation to find possible cuts over areas of low photometric difference – allows for searching over a much smaller set of watershed segments, instead of over the entire set of pixels in the intersection zone. The proposed method presents several advantages. The use of graph cuts over image pairs guarantees the globally optimal solution for each intersection region. The independence of such regions makes the algorithm suitable for parallel implementation. The separated use of the geometric and photometric criteria frees the need for a weighting parameter. Finally, it allows the efficient creation of large mosaics, without user intervention. We illustrate the performance of the approach on image sequences with prominent 3D content and moving objects. 1

Belief propagation (BP) have emerged as powerful tools in the realm of dense stereo computation. However the underlying brightness constancy (BC) assumption of existing methods severely limit the range of their applications. Augmenting BC with gradient constancy (GC) assumption has lead to a more accurate algorithm for optical flow computation. In this paper, these constraints are utilized in the frameworks of BP to broaden the application of stereo vision for 3D reconstruction. Results from experiments with semi-synthetic and real data illustrate that an algorithm incorporating these models generally yields better estimates, where the BC assumption is violated.

Projective homography sits at the heart of many problems in image registration. In addition to many methods for estimating the homography parameters (R.I. Hartley and A. Zisserman, 2000), analytical expressions to assess the accuracy of the transformation parameters have been proposed (A. Criminisi et al., 1999). We show that these expressions provide less accurate bounds than those based on the earlier results of Weng et al. (1989). The discrepancy becomes more critical in applications involving the integration of frame-to-frame homographies and their uncertainties, as in the reconstruction of terrain mosaics and the camera trajectory from flyover imagery. We demonstrate these issues through selected examples.

We investigate the global alignment of some 3-D spatial points, knowing the motion between pairwise nearby positions. A common application is to determine the trajectory of a mobile vision-based system from the scene images acquired along its track. Exploiting redundant measurements and the rigid body motion constraint as the observation model, we apply the mixed adjustment model paradigm to develop recursive estimation algorithms under various scenarios. Results of experiments are given to demonstrate the performance for different noise levels in the observations and the improvements in the position estimation. We also present the results of an experiment with underwater images in the construction of the camera platform trajectory.

We investigate the global alignment of some 3-D spatial points, knowing the motion between pairwise nearby positions. A common application is to determine the trajectory of a mobile vision-based system from the scene images acquired along its track. Exploiting redundant measurements and the rigid body transformation Pj= R(i,J)Pi+t(i,J)as the observation model, we apply the mixed-model least squares estimation paradigm to develop recursive estimation algorithms under various scenarios. Results of experiments are given to demonstrate the performance for different noise levels in the observations and the improvements in the position estimation.

Over the last decade there, has been an increasing interest in developing vision systems and technologies that support the operation of unmanned platforms for positioning, mapping, and navigation. Until very recently, these developments relied on images from standard CCD cameras with a single optical center and limited field of view, making them restrictive for some applications. Panoramic images have been explored extensively in recent years. The particular configuration of interest to our investigation yields a conical view, which is most applicable for airborne and underwater platforms. Instead of a single catadioptric camera (Gluckman, J.M. and Nayar, S.K., 1999; Swaminathan, R. et al., 2001), a combination of conventional cameras may he used to generate images at much higher resolution (Negahdaripour, S. et al., Proc. Oceans, 2001). We derive complete mathematic models of projection and image motion equations for a down-looking conical camera that may be installed on a mobile platform - e.g. an airborne or submersible system for terrain flyover imaging. We describe the calibration of a system comprising multiple cameras with overlapping fields of view to generate the conical view. We demonstrate with synthetic and real data that such images provide better accuracy in 3D visual motion estimation, which is the underlying issue in 3D positioning, navigation, mapping, image registration and photo-mosaicking.

Construction of 3D topographic maps from stereo or monocular video, over coverage areas of kilometer scale, taken by low-altitude airborne platforms is addressed. Two computational frameworks for these two cases are considered, accommodating the online processing of video along the path. In these formulations, stereo disparity information enables the computation of 3D motions and depth maps to be done more readily, however, monocular motion cues provide similar accuracy with more computational steps. The critical issue is to overcome the drift error, which is inherent of the causal frame-to frame motion estimation, as the video frames are acquired. A novel global alignment scheme is proposed, aimed at determining the 3D trajectory most consistent with the estimated 3D motions between pairs of nearby positions. Performance is demonstrated based on experiment with a sequence of 5000 stereo pairs, simulating aerial photographic data from an airborne platform flying at 110 m above (the reference plane of) a 1 km /spl times/ 1 km terrain with 5-65m elevation. Maximum geo-referenced positioning accuracy is roughly 2 m, with elevation error of 1 m or less over 95% of the terrain.

Mosaic-based positioning is a paradigm for the simultaneous construction of a photo-mosaic as a visual map, and its use to achieve accurate positioning. We discuss the application of a novel fusion principle, the so-called an extended covariance intersection (ECI), for addressing the mosaic-based positioning as a data fusion problem. The covariance intersection (CI) principle has been proposed for the fusion of highly correlated data. In contrast to the extended Kalman filter (EKF), the drawback is the conservative nature of the solution, as the extend of correlation becomes insignificant. The primary advantage of ECI, by decomposing the estimates from information sources into both dependent and independent components, is to arrive at improved estimates, neither as over-optimistic as from an EKF, nor as over-conservative as the CI solution. Experiments with real data are presented to evaluate the performance of the proposed ECI-based formulation.

Unmanned vehicles are employed more and more frequently for a range of scientific and commercial undersea applications. However, the critical dependency on a tether link, mainly for the transmission of live images to the surface for command and control, is a significant technological obstacle limiting vehicle maneuverability. The elimination of the tether requires the capability to compress a massive amount of live video data to meet the bandwidth limitations of acoustic telemetry. This paper addresses motion-compensated compression of underwater video imagery and the extraction of the sought-after transformations by the application of the brightness constancy assumption and a generalized dynamic image model for the analysis of time-varying imagery. Two approaches, suitable for automatic vision-based navigation or operator-assisted missions of unmanned submersible vehicles, are considered. In the former case, the 3D motion and position information extracted from the raw data by the vision-based navigation system is used directly to perform the compression. In the latter case, the compression is achieved using the motion and radiometric information extracted from the live and reconstructed images at the surface station. To evaluate the performance of proposed methods, results from experiments with synthetic and real data are presented and compared to compression methods that are based on the brightness constancy model.

A vision system has been developed based on the application of a 3D direct motion estimation algorithm to facilitate autonomous or operated-assisted missions of AUVS and ROVs near the ocean floor. The main capabilities of interest are vision-based vehicle positioning, navigation and trajectory following, as well as mosaicking of sea bed images. The system performance in enabling these functions has been demonstrated on a three-thruster surface vehicle that operates in a 6'/spl times/12'/spl times/6' water tank, with the bottom surface set up to simulate a sea floor environment. Images from a down-look camera installed on the vehicle are digitized and processed on a Windows NT Dual Pentium-200 to estimate the vehicle's motion, which is transmitted via the serial link to the vehicle control system running on a 386 processor. This information is employed to maintain or move to a desired position, follow a specified trajectory, and to construct in (near) real-time a composite image of the scene surface.

Optical flow has been commonly defined as the apparent motion of image brightness patterns in an image sequence. In this paper, we propose a revised definition to overcome shortcomings in interpreting optical flow merely as a geometric transformation field. The new definition is a complete representation of geometric and radiometric variations in dynamic imagery. We argue that this is more consistent with the common interpretation of optical flow induced by various scene events. This leads to a general framework for the investigation of problems in dynamic scene analysis, based on the integration and unified treatment of both geometric and radiometric cues in time-varying imagery. We discuss selected models, including the generalized dynamic image model, for the estimation of optical flow. We show how various 3D scene information are encoded in, and thus may be extracted from, the geometric and radiometric components of optical flow. We provide selected examples based on experiments with real images.

The authors present a real-time vision-based system for automatic positioning and trajectory following, based on a direct method for 3D motion estimation. The spatio-temporal derivatives of the image function, calculated from time-varying imagery, are used to directly calculate the motion and position of the camera. For demonstration, they have implemented the system on a one-degree-of freedom thruster operating in a laboratory water tank. The estimated position information is communicated to the control system, a PID controller, in order to generate the appropriate signal to correct the thruster system's position. The performance of the vision system is demonstrated in selected experiments by comparing results with the data from an optical encoder position sensor.

We propose a direct method for locating the focus of expansion (FOE), based on simple parallel computations in selected regions of the image; each is a circular patch around an estimated FOE. Simple computations allow determining the direction from the estimated to the true FOE. The best estimate of the intersection of the so-called FOE constraint lines for several regions gives the location of the FOE. Some analysis allows us to assign a confidence measure to the information from each local region, in order to give more weighting to the most reliable data. Hence, the FOE can be located with more accuracy, even when the data from various local regions lacks sufficient information. Results of experiments with real images of various texture content are given to demonstrate the performance of the method.

We address the problem of motion recovery for a head-eye system from stereo image sequences. Two types of motions, the translation of the vehicle and the panning motion of the head, are considered. We show how these motions and the depth map of the scene can be estimated directly from the measurements of image gradients and time derivatives in a sequence of stereo images. There is no need to estimate image motion, track a scene feature over time, or establish point correspondences in a stereo image pair. We present the results of various experiments with real scenes.

The authors propose an image motion constraint equation based on a model which allows the brightness of a scene point to vary with time, unlike the case in the brightness constancy model. Using this model, they describe a method for the computation of optical flow and investigate its performance in a variety of conditions involving brightness variations of scene points, due to illumination nonuniformity, light source motion, specular reflection, and/or interreflection. It is shown that in the application of this method, care must be taken in the estimation of image derivatives using finite difference methods to prevent biases in the solution. A simple modification is suggested to overcome the problem. A comparison is made with two other models, including the classical brightness constancy model, through results from experiments with real images.>

A direct method is proposed for locating the focus of expansion, based on simple parallel computations in selected regions of the image; each is a circular patch around an estimated focus of expansion (FOE). Simple computations allow determining the direction from the estimated to the true 400FOE. The best estimate of the intersection of the so-called FOE constraint lines for each region gives the location of the FOE. Some analysis allows a confidence measure to be assigned to the information from each local region, in order to give more weighting to the most reliable data. Hence, the FOE can be located with more accuracy, even when the data from various local regions lack sufficient information.>

The problem of motion recovery for a head-eye system from stereo image sequences is addressed. Two types of motions, the translation of the vehicle and the panning motion of the head, are considered. It is shown how these motions and the depth map can be estimated directly from the measurements of image gradients and time derivatives. There is no need to estimate image motion, track a scene feature over time, or establish point correspondences in a stereo image pair. The results of various experiments with real scenes are presented.>

A particular advantage of an optical stationkeeping system is its ability to use natural rather than man-made beacons. Improvements to previously reported optical flow methods for detecting vehicle motion are presented. Experimental results indicate that an adaptation of Newton-Raphson search combined with the use of a low-noise, high-accuracy camera drastically reduces the number of points at which computations need be done. Experiments with an algorithm which accounts for illumination variations one encounters in undersea environments show significant improvement in the estimation of vehicle motion.>

Using a stereo vision system, the authors show how the translational motion and scene structure can be recovered directly from image gradients and time derivatives. There is no need to estimate or establish correspondences between features across images. The direction of motion is recovered using a procedure which involves minimizing the sum of the squared error of a linear constraint equation over the entire image. The magnitude of the motion is estimated from the stereo disparity. The scene structure is recovered in the form of a depth map using the recovered motion, image gradients, and time derivatives. Experimental results using real images are presented.>

The following problem is investigated: given the position coordinates in two images of all points on an object obtained from two different camera positions, under what conditions can there be more than one interpretation for the shape of the object and the transformation between the coordinate systems at the two camera positions? It is shown that only certain hyperboloids of one sheet and their degeneracies, such as hyperbolic paraboloids, circular cylinders, and intersecting planes, that are viewed from a point on their surface can give rise to an ambiguity. In the case of hyperboloids of one sheet and hyperbolic paraboloids, there can be three possible solutions. In the case of circular cylinders and intersecting planes, there are at most two solutions. The author gives the relationship among the multiple interpretations and determines them all in closed form in terms of the true solution.>

Recovering, from two-dimensional images, certain three-dimensional properties of the scene, such as motion and shape of objects in a scene and their spatial arrangement, is one of the primary goals of a machine vision system. Given a textured scene motion can be recovered rather easily if the structure of the scene, usually in the form of a depth map of the scene, is known. In theory, it is also possible to recover shape if the relative motion between the viewer and objects in the scene is known. The authors study the sensitivity of the solution of the first problem to inaccuracies in the knowledge of the depth map of the scene. First a brightness change constraint equation, is used give closed-form solutions for the motion parameters in two cases: known depth, and small depth variations relative to the absolute distance of the scene from the viewer. They then investigate the robustness of the solution for the motion parameters in terms of the scene structure, modeled as piecewise planar patches. They demonstrate the behavior of the solution with the variations of the orientation of the surface patch being viewed and the size of the field of view. Using a quantitative measure, the show the need for a large field of view to recover motion robustly. The eigenvalue-eigenvector decomposition of a 6*6 matrix allows the authors to analyze some well-known ambiguities in recovering motion.>

In this correspondence, we show how to recover the motion of an observer relative to a planar surface from image brightness derivatives. We do not compute the optical flow as an intermediate step, only the spatial and temporal brightness gradients (at a minimum of eight points). We first present two iterative schemes for solving nine nonlinear equations in terms of the motion and surface parameters that are derived from a least-squares fomulation. An initial pass over the relevant image region is used to accumulate a number of moments of the image brightness derivatives. All of the quantities used in the iteration are efficiently computed from these totals without the need to refer back to the image. We then show that either of two possible solutions can be obtained in closed form. We first solve a linear matrix equation for the elements of a 3 × 3 matrix. The eigenvalue decomposition of the symmetric part of the matrix is then used to compute the motion parameters and the plane orientation. A new compact notation allows us to show easily that there are at most two planar solutions.

In this paper, we derive a closed form solution for recovering the motion of an observer relative to a planar surface directly from image brightness derivatives. We do not compute the optical flow as an intermediate step, only the spatial and temporal intensity gradients at a minimum of 8 points. We solve a linear matrix equation for the elements of a 3×3 matrix. The eigenvalue decomposition of its symmetric part is then used to compute the motion parameters and the plane orientation.