Shape from Polarization (SfP) aims to recover surface normal using the polarization cues of light. The accuracy of existing SfP methods is affected by two main problems. First, the ambiguity of polarization cues partially results in false normal estimation. Second, the widely-used assumption about orthographic projection is too ideal. To solve these problems, we propose the first approach that com-bines deep learning and stereo polarization information to recover not only normal but also disparity. Specifically, for the ambiguity problem, we design a Shape Consistency-based Mask Prediction (SCMP) module. It exploits the inherent consistency between normal and disparity to identify the areas with false normal estimation. We replace the unreliable features enclosed by these areas with new features extracted by global attention mechanism. As to the orthographic projection problem, we propose a novel Viewing Direction-aided Positional Encoding (VDPE) strategy. This strategy is based on the unique pixel-viewing direction encoding, and thus enables our neural network to handle the non-orthographic projection. In addition, we establish a real-world stereo SfP dataset that contains various object categories and illumination conditions. Experiments showed that compared with existing SfP methods, our approach is more accurate. Moreover, our approach shows higher robustness to light variation.

Most existing methods for category-level pose estimation rely on object point clouds. However, when considering transparent objects, depth cameras are usually not able to capture high-quality data, resulting in point clouds with severe artifacts. Without a complete point cloud, existing methods are not applicable to challenging transparent objects. To tackle this problem, we present StereoPose, a novel stereo image based framework for category-level object pose estimation, ideally suited for transparent objects. For a robust estimation from pure stereo images, we develop a pipeline that decouples category-level pose estimation into object size estimation, initial pose estimation, and pose refinement. StereoPose then estimates object pose based on representation in the normalized object coordinate space (NOCS). To address the issue of image content aliasing, we further define a back-view NOCS map for the transparent object. The back-view NOCS aims to reduce the network learning ambiguity caused by content aliasing, and leverage informative cues on the back of the transparent object for more accurate pose estimation. To further improve the performance of the stereo framework, StereoPose is equipped with a parallax attention module for stereo feature fusion and an epipolar loss for improving the stereo-view consistency of network predictions. Extensive experiments on the public TOD dataset demonstrate the superiority of the proposed StereoPose framework for category-level 6D transparent object pose estimation. Code and demos will be available on the project homepage: www.cse.cuhk.edu.hk/~kaichen/stereopose.html.

Three-dimensional (3D) reconstruction of dynamic objects has broad applications, including object recognition and robotic manipulation. However, achieving high-accuracy reconstruction and robustness to motion simultaneously is a challenging task. In this paper, we present a novel method for 3D reconstruction of dynamic objectS, whose main features are as follows. Firstly, a structured-light multiplexing method is developed that only requires 3 patterns to achieve high-accuracy encoding. Fewer projected patterns require shorter image acquisition time, thus, the object motion is reduced in each reconstruction cycle. The three patterns, i.e. spatial-temporally encoded patterns, are generated by embedding a specifically designed spatial-coded texture map into the temporal-encoded three-step phase-shifting fringes. A temporal codeword and three spatial codewords are extracted from the composite patterns using a proposed extraction algorithm. The two types of codewords are utilized separately in stereo matching: the temporal codeword ensures the high accuracy, while the spatial codewords are responsible for removing phase ambiguity. Secondly, we aim to eliminate the reconstruction error induced by motion between frames abbreviated as motion induced error (MiE). Instead of assuming the object to be static when acquiring the 3 images, we derive the motion of projection pixels among frames. Using the extracted spatial codewords, correspondences between different frames are found, i.e. pixels with the same codewords are traceable in the image sequences. Therefore, we can obtain the phase map at each image-acquisition moment without being affected by the object motion. Then the object surfaces corresponding to all the images can be recovered. Experimental results validate the high reconstruction accuracy and precision of the proposed method for dynamic objects with different motion speeds. Comparative experiments show that the presented method demonstrates superior performance with various types of motion, including translation in different directions and deformation.

Three-dimensional reconstruction of dynamic objects is important for robotic applications, for example, the robotic recognition and manipulation. In this paper, we present a novel 3D surface reconstruction method for moving objects. The proposed method combines the spatial-multiplexing and time-multiplexing structured-light techniques that have advantages of less image acquisition time and accurate 3D reconstruction, respectively. A set of spatial-temporal encoded patterns are designed, where a spatial-encoded texture map is embedded into the temporal-encoded three-step phase-shifting fringes. The specifically designed spatial-coded texture assigns high-uniqueness codeword to any window on the image which helps to eliminate the phase ambiguity. In addition, the texture is robust to noise and image blur. Combining this texture with high-frequency phase-shifting fringes, high reconstruction accuracy would be ensured. This method only requires 3 patterns to uniquely encode a surface, which facilitates the fast image acquisition for each reconstruction step. A filtering stereo matching algorithm is proposed for the spatial-temporal multiplexing method to improve the matching reliability. Moreover, the reconstruction precision is further enhanced by a correspondence refinement algorithm. Experiments validate the performance of the proposed method including the high accuracy, the robustness to noise and the ability to reconstruct moving objects.

Precise 3-D surface reconstruction plays an important role in automated manipulation, industrial inspection, robotics, and so on. In this article, we present a novel 3-D surface reconstruction framework for stereo vision systems assisted with structured light projection. In the framework, a multistep matching scheme is proposed to establish a reliable correspondence between image pairs with high computation efficiency and accuracy. The successive matching steps can find the most precise correspondence through a step-by-step filtering procedure. To further enhance the precision, a correspondence refinement algorithm is presented. Phase maps with different frequencies are utilized as the code words for the multistep matching due to their high encoding accuracy and robustness to noise. This method does not require phase unwrapping or projector calibration, which improves the reconstruction precision and simplifies the operation. Selection strategies for the number of matching steps, the pattern frequencies, and the matching threshold are proposed. Furthermore, various 3-D reconstruction experiments are conducted using the proposed framework. Comparative experiments verify the advantages of the proposed framework compared with existing 3-D reconstruction methods regarding the accuracy and precision. The adaptability to scenarios with different motion speeds is demonstrated. Robustness and limitations of the framework are also revealed by conducting experiments in challenging scenarios. Note to Practitioners-This article is motivated by the precise 3-D surface reconstruction problem in automated robotic systems. In different scenarios, such as the reconstruction of the static objects or moving objects, the errors induced by sensor noise and motion should be taken into consideration. To enhance the measurement precision under these occasions, selection of pattern number and fringe frequencies has been a problem. To overcome these problems, this article proposes a novel framework for active stereo 3-D surface reconstruction. The framework utilizes multifrequency phase-shifting fringes to encode the reconstructed target. Then, a multistep matching method filters the candidates step by step to obtain the most precise corresponding pixel and avoid noise error accumulation. A refinement method is introduced to further improve the precision. Selection strategies of the number of matching steps, the fringe frequencies, and matching thresholds enable the 3-D reconstruction framework to be utilized on different occasions. In applications, limitations of the proposed method should be noted.

Three-dimensional vision plays an important role in robotics. In this paper, we present a 3D surface reconstruction scheme based on combination of stereo matching and pattern projection. A two-step matching scheme is proposed to establish reliable correspondence between stereo images with high computation efficiency and accuracy. The first step (coarse matching) can quickly find the correlation candidates, and the second step (precise matching) is responsible for determining the most precise correspondence within the candidates. Two phase maps serve as codewords and are utilized in the two-step stereo matching, respectively. The phase maps are derived from phase-shifting patterns to provide robustness to the background noises. Only five patterns are required, which reduces the image acquisition time. Moreover, the precision is further enhanced by applying a correspondence refinement algorithm. The precision and accuracy are validated by experiments on standard objects. Furthermore, various experiments are conducted to verify the capability of the proposed method, which includes the complex object reconstruction, the high-resolution reconstruction, and the occlusion avoidance. The real-time experimental results are also provided.

In this paper, we propose a novel 3D laparoscopic imaging system based on stereo-photogrammetry which is assisted by projecting patterns on the tissue surface. The proposed laparoscopic imaging system has three optic channels, two of which are responsible for stereo vision feedback and the other one is used for coded structured patterns projection. The projected patterns provide the robustness to homogeneous tissue surface since they add more features that can be relied on in the stereo matching. Image fiber bundles (100k pixels) and Gradient-index (GRIN) lenses are utilized to facilitate the remote image acquisition and miniaturization of the laparoscopic probe. Moreover, we adopt a digital micromirror device (DMD) and high-speed cameras to achieve fast pattern switching (up to 4 kHz) and high frame rate image acquisition. The system configuration allows for implementation of the time multiplexing pattern codification strategy in the 3D laparoscopic imaging system to enhance the reliability and resolution of the 3D surface reconstruction. A prototype is established, and various experiments are conducted. Comparative experimental results prove the advantages of our system design. The static and dynamic 3D reconstruction results validate the performance of the proposed 3D laparoscopic imaging system quantitatively and qualitatively.