Augmented reality (AR) assembly guidance systems can help users quickly master the assembly process of unfamiliar objects. However, it is very difficult for practical use without a thorough understanding of users' intentions. Furthermore, existing approaches still struggle to handle complex hand-part interactions and nonlinear assembly steps, and achieving intuitive and real-time guidance remains challenging. To address these issues, we propose an intention-driven camera-projector AR assembly guidance system (IntentionAR) that integrates online user intention inference with a finite-state assembly machine. The intention module recognizes interaction actions and infers the target object, enabling feedback before assembly begins, while the finite state machine (FSM) manages step progression. Using a camera-projector device, the system highlights candidate parts and, based on inferred intention, flags correct/incorrect selections to provide real-time guidance. As the assembly progresses, the models used for spatial registration and visualization switch dynamically to accommodate the nonlinear workflow. Experiments and user studies show that the system delivers a more robust AR interaction experience and improves assembly efficiency. A free copy of this paper and all supplemental materials, as well as project assets and source code, will be available at the project website.

The existing methods for 6D pose estimation based on RGB-D employ RGB images and observed point cloud derived from depth maps as input, then concurrently predicting both rotation and translation. However, rotation and translation possess distinct characteristics and scale ranges, and their simultaneous prediction can lead to mutual influence in the network parameter space. Additionally, the observed point cloud are susceptible to systematic noise and partial data loss, presenting challenges for the network to capture comprehensive object features. To address these issues, we propose the Semi-Decoupled 6D pose estimation via multi-modal feature fusion (SD6D). SD6D comprises a Multi-Modal Fusion Module and a Semi-Decoupled Prediction Module. The former dynamically fuses different modal data (RGB, depth, CAD model) based on their inter-modality correlations, aiding in establishing 2D-3D correspondences and addressing issues stemming from systematic noise and partial data loss. The latter semi-decouples the prediction of rotation and translation, predicting them separately based on their distinct characteristics. We conducted experiments on two popular benchmark datasets, which prove the superiority of our method.

Category-level pose tracking methods can continuously track the pose of objects without requiring any prior knowledge of the specific shape of the tracked instance. This makes them advantageous in augmented reality and virtual reality applications. The key challenge is how to train neural networks to accurately predict the poses of objects they have never seen before and exhibit strong generalization performance. We propose a novel category-level 6D pose tracking method Corr-Track, which is capable of accurately tracking objects belonging to the same category from depth video streams. Our approach utilizes direct soft correspondence constraints to train a neural network, which estimates bidirectional soft correspondences between sparsely sampled point clouds of objects in two frames. We first introduce a soft correspondence matrix for pose tracking tasks and establish effective constraints through direct spatial point-to-point correspondence representations in the sparse point cloud correspondence matrix. We propose the "point cloud expansion" strategy to address the "point cloud shrinkage" problem resulting from soft correspondences. This strategy ensures that the corresponding point cloud accurately reproduces the shape of the target point cloud, leading to precise pose tracking results. We evaluated our approach on the NOCS-REAL275 and Wild6D dataset and observed superior performance compared to previous methods. Additionally, we conducted cross-category experiments that further demonstrated its generalization capability.

Hand-eye calibration estimates the pose of a camera relative to a robot, which is a fundamental problem for visually guided robots, especially for dynamic object grasping. Most methods use 2D fiducial markers with distinctive visual features and require pre-calibration for accurate calibration, which can not work online. In this paper, we propose a novel hand-eye calibration method based on the natural 3D object, which can work online and automatically even if the object is textureless or weakly textured. We first propose a Pose Refinement Network (PR-Net) to improve the accuracy of 3D object tracking. Then we build a 3D convergence point constraint based on the multi-view information with the accurate object pose to adjust the object position. Finally, we optimize the hand-eye pose by the closed-loop constraint with the optimized object position, solving the problem that is easy to fall into a local minimum. The experiments show that the average error of our hand-eye calibration method is 1.20 degrees and 23.18 mm. The results achieve state-of-the-art by using the working object to realize the online hand-eye calibration.

Template-based 3D object tracking still lacks a high-precision benchmark of real scenes due to the difficulty of annotating the accurate 3D poses of real moving video objects without using markers. In this paper, we present a multi-view approach to estimate the accurate 3D poses of real moving objects, and then use binocular data to construct a new benchmark for monocular textureless 3D object tracking. The proposed method requires no markers, and the cameras only need to be synchronous, relatively fixed as cross-view and calibrated. Based on our object-centered model, we jointly optimize the object pose by minimizing shape reprojection constraints in all views, which greatly improves the accuracy compared with the single-view approach, and is even more accurate than the depth-based method. Our new benchmark dataset contains 20 textureless objects, 22 scenes, 404 video sequences and 126K images captured in real scenes. The annotation error is guaranteed to be less than 2mm, according to both theoretical analysis and validation experiments. We reevaluate the state-of-the-art 3D object tracking methods with our dataset, reporting their performance ranking in real scenes. Our BCOT benchmark and code can be found at https://ar3dv.github.io/BCOT-Benchmark/.