As a complicated computer vision task, goal of Semantic Scene Completion(s) is to predict each voxels’ occupancy and corresponding semantic category of 3D scene. In order to reduce computation burden brought by 3D convolution, recently, some methods transform voxelized scenes into point clouds through removing visible empty voxels. However, due to the inherent feature imbalance among the valid "voxel-points", reconstruction quality of these methods is limited. In this paper, we propose a novel point-based SSC to solve the dilemma. Firstly, we design a novel Surface-Attention module to compensate shortage of feature on voxels behind observed surfaces. Meanwhile, correlation between adjacent points from the same category is consolidated through Soft-Semantic Transformer layer. Experiment results on NYU and NYUCAD datasets demonstrate superiority of our method both intuitively and quantitively. Our code is available at https://github.com/furuochong.

Semantic scene reconstruction from sparse and incomplete point clouds is a critical task for point scene understanding. It aims to recognize semantic labels for objects and recover their complete shapes as meshes. Existing methods often fail to realize high-quality instance reconstruction due to inadequate shape representation and underutilization of proposal point clouds. To address these issues, we optimize the previous BSP/occupancy-to-mesh reconstruction framework to points-to-mesh and accomplish multi-level utilization of proposals. We chose point cloud as the representation of completion to reduce the difficulty of restoring curved shallow parts. Benefiting from the optimization, we can match and merge proposal point clouds with the restored ones, avoiding missing parts existing in inputs. We design an effective pose normalization module to extract point-based features from normalized proposals, which are fused with features extracted from voxelized proposals, avoiding the detailed geometry lost in voxelization and enhancing the reconstruction's robustness to different input postures. The suitable points-to-mesh reconstruction framework and full utilization of proposals make our method improve reconstruction results efficiently. Detailed experiments on the challenging ScanNet dataset of the semantic scene reconstruction benchmark show that our network outperforms state-of-the-art methods in both completion and mapping metrics.

Conventional point cloud completion approaches seldom consider object poses when generating shapes, which could make their networks be vulnerable to pose variations. A possible solution, rotation augmentation in training, would degrade prediction accuracy and cannot essentially ensure output stability over different poses. Therefore, in this paper, we introduce a new generative network that is equivariant-by-construction on SO(3) rotation group. It restores complete shapes following the coarse-to-fine fashion with attention-based vector neurons. We design several modules in our network: a Dynamic Projection Kernel to predict spatial points based on constrained projections on three dynamic coordinates, a Transformer-based Structure Awareness module to analyze local structures, a Local Geometry Propagation Pipeline to extract and fuse point-wise features, and an equivariant Folding Kernel to refine and upsample coarse point clouds. Compared with existing approaches, our network achieves competitive performances on both synthetic and real-world objects1.

Partial observation of indoor scenes (single-viewed RGB-D) carries insufficient spatial information for complex tasks such as autonomous navigation and virtual reality, thus many learning-based methods are proposed to realize semantic scene completion (SSC) from single-viewed input. However, most of them only extract scene-level features of input to generate output, which might lose details. In this paper, a new method that fully utilizes both instance-level and scene-level features is proposed. Firstly, an object detection module is pre-trained to localize indoor objects. Secondly, coarse completion result is obtained from scene-level feature using an encoder-decoder structure. Finally, based on the pre-trained bounding boxes, coarse completion result is refined using a geometric refinement module. Our network's performance is evaluated on both real and synthetic datasets. The results demonstrate that our network is able to reconstruct indoor scenes with more geometric details, get clearer boundaries between instances and outperform most existing SSC methods both intuitively and quantitatively.

Point clouds generated from real-world scanning are always incomplete and ununiformly distributed, which would cause structural losses in 3D shape representations. Therefore, a learning-based method is introduced in this paper to repair partial point clouds and restore complete shapes of target objects. First, we sample several local regions of inputs, encode their features and fuse them with independently extracted global features. Second, we establish a graph to connect all local features together, and then implement convolution with multi-head attention on the graph. Graph attention mechanism enables each local feature vector to search across the regions and selectively absorb other local features based on their relationships in high-dimensional feature space. Third, we design a coarse decoder to collect cross-region features from the graph and generate skeletons of complete point clouds, and a folding-based decoder is leveraged to generate final point clouds with high resolution. Our network is trained on six categories of objects from the ModelNet dataset, its performance is compared with several existing methods, the results show that our network is able to generate dense complete point cloud with the highest accuracy.