While recent advancements have demonstrated remarkable progress in general 3D shape generation, the challenge of automatically generating wearable 3D assets remains largely unexplored. To address this gap, we present BAG - a Body-aligned Asset Generation method that produces 3D wearable assets which can be automatically fitted onto given 3D human bodies. This is achieved by controlling the 3D generation process using human body shape and pose information. Specifically, we first construct a general single-image-to-consistent-multi-view diffusion model, and train it on the large-scale Objaverse dataset to ensure diversity and generalizability. We then train a body-conditioned multi-view ControlNet to guide the generator toward producing body-aligned multi-view images. The control signal leverages multi-view 2D projections of the target human body, where pixel values represent the XYZ coordinates of the body surface in a canonical space. The resulting body-conditioned multi-view diffusion outputs body-aligned images, which are subsequently fed into a native 3D diffusion model to reconstruct the 3D shape of the asset. Finally, we recover the similarity transformation using multi-view silhouette supervision and mitigate asset-body penetration using physics-based simulation, ensuring accurate asset fitting onto the target body. Experimental results demonstrate that our method significantly outperforms existing approaches in terms of prompt adherence, shape diversity, and shape quality.

Large Reconstruction Models have made significant strides in the realm of automated 3D content generation from single or multiple input images. Despite their success, these models often produce 3D meshes with geometric inaccuracies, stemming from the inherent challenges of deducing 3D shapes solely from image data. In this work, we introduce a novel framework, the Large Image and Point Cloud Alignment Model (LAM3D), which utilizes 3D point cloud data to enhance the fidelity of generated 3D meshes. Our methodology begins with the development of a point-cloud-based network that effectively generates precise and meaningful latent tri-planes, laying the groundwork for accurate 3D mesh reconstruction. Building upon this, our Image-Point-Cloud Feature Alignment technique processes a single input image, aligning to the latent tri-planes to imbue image features with robust 3D information. This process not only enriches the image features but also facilitates the production of high-fidelity 3D meshes without the need for multi-view input, significantly reducing geometric distortions. Our approach achieves state-of-the-art high-fidelity 3D mesh reconstruction from a single image in just 6 seconds, and experiments on various datasets demonstrate its effectiveness.

We present BlockFusion, a diffusion-based model that generates 3D scenes as unit blocks and seamlessly incorporates new blocks to extend the scene. BlockFusion is trained using datasets of 3D blocks that are randomly cropped from complete 3D scene meshes. Through per-block fitting, all training blocks are converted into the hybrid neural fields: with a tri-plane containing the geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the signed distance values. A variational auto-encoder is employed to compress the tri-planes into the latent tri-plane space, on which the denoising diffusion process is performed. Diffusion applied to the latent representations allows for high-quality and diverse 3D scene generation. To expand a scene during generation, one needs only to append empty blocks to overlap with the current scene and extrapolate existing latent tri-planes to populate new blocks. The extrapolation is done by conditioning the generation process with the feature samples from the overlapping tri-planes during the denoising iterations. Latent tri-plane extrapolation produces semantically and geometrically meaningful transitions that harmoniously blend with the existing scene. A 2D layout conditioning mechanism is used to control the placement and arrangement of scene elements. Experimental results indicate that BlockFusion is capable of generating diverse, geometrically consistent and unbounded large 3D scenes with unprecedented high-quality shapes in both indoor and outdoor scenarios.

A reliable and accurate visual object 6DoF pose tracking system for texture-less objects plays an important role in various fields of modern industry hence it has been a hot research topic for decades. Traditional feature-based pose tracking methods require rich feature points on objects, and it cannot handle texture-less objects. To tackle this problem, the paper proposes a novel edge-based method for continuous 6DOF pose tracking of texture-less objects. The pose of the object is estimated by minimizing the matching error between extracted image edges and re-projected CAD model edges. The matching error is represented using the Directional Chamfer Matching (DCM) Tensor. Compared with previous methods, the proposed system improves the overall performance of pose tracking system in two ways. Firstly, the method proposes an analytical mathematic model in the optimization process; Secondly, the method proposes an adaptive edge point weighting algorithm to tackle the occlusion or edge weakness problem. Both methods help improve the accuracy and robustness of the pose estimation system. With the benefit of GPU acceleration on DCM Tensor calculation the proposed method could run in real time. Extensive experiments are conducted both on public datasets and CG-rendered synthetic datasets to validate the accuracy and robustness of the proposed algorithm against other state-of-the-art methods.

Traditional mapping modules in Visual Simultaneous Localization and Mapping (i.e. Visual SLAM) systems can only estimate 3D information of isolated sparse or semi-dense feature points. But there are lots of object instances in the environments which geometric information can be utilized to enhance the quality of mapping and localization. Hence, it is required for the Visual SLAM system to utilize high-dimensional features like object instances or structural lines in mapping and localization. To meet the gap between the above requirements and the traditional implementation of Visual SLAM systems, we present in this paper a novel Visual SLAM method that can effectively utilize texture-less object instances for mapping and localization. The proposed Visual SLAM method includes newly designed feature extraction, matching, localization and mapping modules, which jointly use object features and point features to estimate camera 6-DOF poses and do richer map construction. A group of organized raster points is used to represent objects during feature matching and pose estimation process in the proposed Visual SLAM pipeline. Owing to the object feature fusion in the co-visibility graph it could conduct scale aware bundle adjustments to reduce accumulated error. The advantages of proposed Visual SLAM method are demonstrated through experiments conducted both on synthetic datasets and real-world datasets.

Facial landmark localization is a crucial initial step Driver Inattention Monitoring. The aim of this paper is to localize driver facial landmarks across large rotation, say [-90°, +90°] in yaw rotation, to cope with real driving conditions. The paper proposes a flexible pipe-line for creating automatically labeled face image to supply wanted dataset. The benefits of CG (Computer Graphics) techniques such as 3D face modelling and morphing, photorealistic rendering and ground truth generation are utilized. To the best of our knowledge this is the first time to combine CG rendering and automatic ground truth labelling techniques with face landmark localization algorithms. The effectiveness of the CG rendered data is proved by cross validation with Multi-PIE dataset. Landmark localization across large rotation is obtained by a system simply integrating the off the-shelves algorithms and trained with the CG rendered data. The experiments of the implemented system on Multi-PIE and real persons show that it could localize facial landmarks across large rotation accurately and in real time.

The state of art Visual SLAM is going from sparse feature to semi-dense feature to provide more information for environment perception, whereas the semi-dense methods often suffer from inaccurate depth map estimation and are easy to become instable for some real-world scenarios. The paper proposes to extend the ORB-SLAM2 framework, which is a robust sparse feature SLAM system tracking camera motion with map maintenance and loop closure, by introducing the unified spherical camera model and the semi-dense depth map. The unified spherical camera model fits the omnidirectional camera well, therefore the proposed Visual SLAM system could handle fisheye cameras which are commonly installed on modern vehicles to provide larger perceiving region. In addition to the sparse corners features the proposed system also utilizes high gradient regions as semi-dense features, thereby providing rich environment information. The paper presents in detail how the unified spherical camera model and the semi-dense feature matching are fused with the original SLAM system. Both accuracies of camera tracking and estimated depth map of the proposed SLAM system are evaluated using real-world data and CG rendered data where the ground truth of the depth map is available.

This paper proposes a method that realizes moving object detection (MOD) and static obstacle detection (SOD) in real time utilizing the fisheye cameras of the around viewing system (AVM). The topview of the AVM is used to calculate the vehicles movement between two frames using homograph estimation. Image features are detected and tracked evenly using cell detection technique. Then the features are projected onto the unit sphere of the fisheye camera model and the epipolar constraint is used to discriminate moving features from static ones. Moving features are clustered into different objects according to their position and orientation. 3D position of the static feature is calculated using triangular principle and potential obstacle is defined as objects above the ground and near the vehicle. The experiment shows the proposed method is robust and accurate for MOD and SOD.