In this work, we aim to improve the 3D reasoning ability of Transformers in multi-view 3D human pose estimation. Recent works have focused on end-to-end learning-based transformer designs, which struggle to resolve geometric information accurately, particularly during occlusion. In-stead, we propose a novel hybrid model, MVGFormer, which has a series of geometric and appearance modules organized in an iterative manner. The geometry modules are learning-free and handle all viewpoint-dependent 3D tasks geometrically which notably improves the model's gener-alization ability. The appearance modules are learnable and are dedicated to estimating 2D poses from image signals end-to-end which enables them to achieve accurate es-timates even when occlusion occurs, leading to a model that is both accurate and generalizable to new cameras and geometries. We evaluate our approach for both in-domain and out-of-domain settings, where our model consistently outperforms state-of-the-art methods, and especially does so by a significant margin in the out-of-domain setting. We will release the code and models: https://github.com/XunshanMan/MVGFormer.

3D object-level mapping is a fundamental problem in robotics, which is especially challenging when object CAD models are unavailable during inference. We propose a framework that can reconstruct high-quality object-level maps for unknown objects. Our approach takes multiple RGB-D images as input and outputs dense 3D shapes and 9-DoF poses (including 3 scale parameters) for detected objects. The core idea is to leverage a learnt generative model for a category of object shapes as priors and to formulate a probabilistic, uncertainty-aware optimization framework for 3D reconstruction. We derive a probabilistic formulation that propagates shape and pose uncertainty through two novel loss functions. Unlike current state-of-the-art approaches, we explicitly model the uncertainty of the object shapes and poses during our optimization, resulting in a high-quality object-level mapping system. Moreover, the estimated shape and pose uncertainties, which we demonstrate can accurately reflect the true errors of our object maps, can be useful for downstream robotics tasks such as active vision. We perform extensive evaluations on indoor and outdoor real-world datasets, achieving substantial improvements over state-of-the-art methods. Our code is available at https://github.com/TRAILab/UncertainShapePose.

3D object reconstruction is important for semantic scene understanding. It is challenging to reconstruct detailed 3D shapes from monocular images directly due to a lack of depth information, occlusion and noise. Most current methods generate deterministic object models without any awareness of the uncertainty of the reconstruction. We tackle this problem by leveraging a neural object representation which learns an object shape distribution from large dataset of 3d object models and maps it into a latent space. We propose a method to model uncertainty as part of the representation and define an uncertainty-aware encoder which generates latent codes with uncertainty directly from individual input images. Further, we propose a method to propagate the uncertainty in the latent code to SDF values and generate a 3d object mesh with local uncertainty for each mesh component. Finally, we propose an incremental fusion method under a Bayesian framework to fuse the latent codes from multi-view observations. We evaluate the system in both synthetic and real datasets to demonstrate the effectiveness of uncertainty-based fusion to improve 3D object reconstruction accuracy.

Plane features can be used to reduce drift errors in SLAM systems, especially in indoor environments. It is easy and efficient to extract planes from a dense point cloud, which is commonly generated from a RGB-D camera or a 3D lidar. But when using a stereo camera, it is hard to compute dense point clouds accurately or efficiently. In this paper, we propose a novel method to compute plane parameters using intersecting lines, which are extracted from stereo images. Plane features are commonly extracted from the surface of man-made objects or structures, which have regular shapes and straight edge lines. In three dimensions, two intersecting lines determine a unique plane. Therefore, we extract line segments from both left and right images of a stereo camera. By stereo matching, we compute lines’ endpoints and direction vectors, and then a plane from two intersecting lines is calculated. We discard inaccurate plane features in the frame tracking. Adding such plane features in the stereo SLAM system reduces drift errors and refines the performance. Finally, we build a global map consisting of both points and planes, which can reflect real scene structures. We test our proposed system on public datasets and demonstrate its accurate estimation results, compared with state-of-the-art SLAM systems. To benefit the research of plane-based SLAM, we release our codes at https://github.com/fishmarch/Stereo-Plane-SLAM.