Understanding surface material properties is crucial for enhancing indoor robot perception and indoor digital twinning. However, not all sensor modalities typically employed for this task are capable of reliably capturing detailed surface material characteristics. By analyzing the reflected RF signal from a mmWave radar sensor, it is possible to extract information about the reflective material and its composition from a certain surface. We introduce a mmWave MIMO FMCW radar-based surface material classifier SMCNet, employing a complex-valued Convolutional Neural Network (CNN) and complex radar IQ signal input for classifying indoor surface materials. While current radar-based material estimation approaches rely on a fixed sensing distance and constrained setups, our approach incorporates a setup with multiple sensing distances. We trained SMCNet using data from three distinct distances and subsequently tested it on these distances, as well as on two more unseen distances. We reached an overall accuracy of 99.12-99.53% on our test set. Notably, range FFT pre-processing improved accuracy on unknown distances from 25.25% to 58.81% without re-training.

Retrieving CAD models for real-world object scans enhances object-level mapping, providing a nuanced spatial understanding crucial for precise interactions in robotics or mixed reality. Commonly, CAD model retrieval is performed by matching features learned in Euclidean space. However, learning discriminative features in Euclidean space faces significant challenges, primarily due to its flat nature and the wide variety of CAD models with different levels of detail. To address the limitations of Euclidean space and improve CAD model retrieval, this paper introduces HypCAD, a contrastive learning framework in hyperbolic space. We present a novel geometry-enhanced hyperbolic distance and utilize a three-component contrastive learning loss to learn hyperbolic feature representations for the CAD model retrieval task. We demonstrate HypCAD’s superior retrieval accuracy through comparisons with baseline contrastive learning methods on both the synthetic ShapeNet dataset and the real-world Scan2CAD dataset.

Point cloud registration is a crucial task for robotics and mixed reality applications, serving as a foundational component for problems such as 3D reconstruction and localization. Arbitrary poses and real-world artifacts, including noise and occlusions, increase registration uncertainty and limit the performance of current point cloud registration algorithms. This paper proposes a novel, sampling-free uncertainty quantification and refinement method for point cloud registration, termed EQUR. To consistently predict uncertainty with high robustness, we extract equivariant point features, from which we regress an uncertainty score, enabling robust quantification of registration uncertainty. Subsequently, we leverage the estimated registration uncertainty as an auxiliary input to enhance the prediction of transformation refinement terms. We employ an introspective learning strategy to train EQUR based on the errors of a baseline registration model. Through quantitative and qualitative analyses on synthetic ShapeNet and real-world ScanObjectNN datasets, we showcase the effectiveness of EQUR, demonstrating both high accuracies in uncertainty quantification and uncertainty-aided refinement of point cloud registration.

In recency, neural signed distance fields have become more popular for reconstructing 3D indoor environments. While great improvements have been made due to missing incident radiance and materials in the surface estimation, current methods cannot reconstruct high-quality surfaces. To address this issue, we propose Neural Painted Radiosity Fields (NPRF), consisting of Neural Radiosity Fields for volumetric surface representation and Neural Painted Scenes for novel view synthesis. Neural Radiosity Fields combine the radiative transfer equation with neural radiosity to estimate 3D surfaces, thus leveraging raytracing to improve the volumetric representation. Neural Painted Scenes employs sparsification and projection of 3D points into 2D images in conjunction with a generative, context-aware inpainting network to produce high-quality novel views. We show that NPRF leads to overall improvements in F-score on the popular ScanNet dataset. Finally, we show that NPRF improves novel view synthesis by a significant margin, giving improvements of up to 25% on PSNR, 53% on LPIPS, and 3% on SSIM.

Recently, SO(3)-equivariant methods have been explored for 3D reconstruction via Scan-to-CAD. Despite significant advancements attributed to the unique characteristics of 3D data, existing SO(3)-equivariant approaches often fall short in seamlessly integrating local and global contextual information in a widely generalizable manner. Our contributions in this paper are threefold. First, we introduce Spherical Patch Fields, a representation technique designed for patch-wise, SO(3)-equivariant 3D point clouds, anchored theoretically on the principles of Spherical Gaussians. Second, we present the Patch Gaussian Layer, designed for the adaptive extraction of local and global contextual information from resizable point cloud patches. Culminating our contributions, we present Learnable Spherical Patch Fields (DeepSPF) – a versatile and easily integrable backbone suitable for instance-based point networks. Through rigorous evaluations, we demonstrate significant enhancements in Scan-to-CAD performance for point cloud registration, retrieval, and completion: a significant reduction in the rotation error of existing registration methods, an improvement of up to 17\% in the Top-1 error for retrieval tasks, and a notable reduction of up to 30\% in the Chamfer Distance for completion models, all attributable to the incorporation of DeepSPF.

Given its wide application in robotics, point cloud registration is a widely researched topic. Conventional methods aim to find a rotation and translation that align two point clouds in 6 degrees of freedom (DoF). However, certain tasks in robotics, such as category-level pose estimation, involve non-uniformly scaled point clouds, requiring a 9DoF transform for accurate alignment. We propose HEGN, a novel equivariant graph neural network for 9DoF point cloud registration. HEGN utilizes equivariance to rotation, translation, and scaling to estimate the transformation without relying on point correspondences. Based on graph representations for both point clouds, we extract equivariant node features aggregated in their local, cross-, and global context. In addition, we introduce a novel node pooling mechanism that leverages the cross-context importance of nodes to pool the graph representation. By repeating the feature extraction and node pooling, we obtain a graph hierarchy. Finally, we determine rotation and translation by aligning equivariant features aggregated over the graph hierarchy. To estimate scaling, we leverage scale information in the vector norm of the equivariant features. We evaluate the effectiveness of HEGN through experiments with the synthetic ModelNet40 dataset and the real-world ScanObjectNN dataset. The results show the superior performance of HEGN in 9DoF point cloud registration and its competitive performance in conventional 6DoF point cloud registration.

Visual SLAM is an essential tool in diverse applications such as robot perception and extended reality, where feature-based methods are prevalent due to their accuracy and robustness. However, existing methods employ either hand-crafted or solely learnable point features and are thus limited by the feature attributes. In this paper, we propose incorporating hybrid point features efficiently into a single system. By integrating hand-crafted and learnable features, we seek to capitalize on their complementary attributes in both key-point identification and descriptor expressiveness. To this purpose, we design a pre-processing module, which includes extraction, inter-class processing, and post-processing of hybrid point features. We present an efficient matching approach to exclusively perform the data association within the same class of features. Moreover, we design a Hybrid Bag-of-Words (H-BoW) model to deal with hybrid point features in matching and loop-closure-detection. By integrating the proposed framework into a modern feature-based system, we introduce HPF-SLAM. We evaluate the system on EuRoC-MAV and TUM-RGBD benchmarks. The experimental results show that our method consistently surpasses the baseline at comparable speed.

3D scene- and object-level scans typically result in sparse and incomplete point clouds. Since dense point clouds of high quality are essential for the 3D reconstruction process, a promising approach is to improve the scan quality by point cloud completion. In this paper, we present COCCA, an extension of point cloud completion networks for scan-to-CAD use cases. The proposed extension is based on cross-attention of features extracted from a scan with rotation-, translation-, and scale-invariant features extracted from a sampled CAD point cloud. With the proposed cross-attention operation, we improve the learning of scan features and the subsequent decoding to a complete shape. We demonstrate the effectiveness of COCCA on the ShapeNet dataset in quantitative and qualitative experiments. COCCA improves the overall completion performance of point cloud completion networks by up to 11.8% for Chamfer Distance and up to 2.2% for F-Score. Our qualitative experiments visualize how COCCA completes point clouds with higher geometric detail. In addition, we demonstrate how completion by COCCA improves the point cloud registration task required for scan-to-CAD alignment.

Scan-to-CAD-based 3D reconstruction of indoor environments has become increasingly more popular in recent years. The inherent structure of Scan-to-CAD consists of object detection, model retrieval, and alignment. Therefore, a variety of metrics are required to assess these three aspects. This can lead to ambiguous evaluation results and incorrect quality assumptions. To impede the problem of incorrect evaluation, we introduce S2CMAF, a multi-method assessment fusion approach for Scan-to-CAD pipelines. S2CMAF merges several metrics used in evaluating these pipelines into one unique quality score. We show that S2CMAF significantly improves the correlation between Scan-to-CAD results and the ground truth, compared to the conventionally used Scan2CAD benchmark. Additionally, we train S2CMAF using different optimization techniques and demonstrate the advantages of our approach on real-world data.