Abstract Jointly performing semantic and instance segmentation of 3D point cloud remains a challenging task. In this work, a novel framework called joint 3D semantic‐instance segmentation via multi‐scale Semantic Association and Salient point clustering Optimization was proposed to tackle this problem. Inspired by the inherent correlation among objects in semantic space, a Multi‐scale Semantic Association (MSA) module to explore the constructive effect of the context information for semantic segmentation is designed. For instance, segmentation, different from previous works utilising clustering only in inference procedure, a Salient Point Clustering Optimization (SPCO) module is put forward to introduce the clustering algorithm into the training phase, which impels the network to focus on points that are difficult to be distinguished. Furthermore, affected by the inherent structure of indoor scenes, the problem of uneven distribution of categories has rarely been considered in the previous work, but it significantly limits the performance of 3D scene perception. To address the issue, an adaptive Water Filling Sampling (WFS) algorithm to balance the category distribution of training data is presented. Extensive experiments on a variety of changing datasets show that the authors’ method outperforms the state‐of‐the‐art methods in both tasks of semantic segmentation and instance segmentation.

We propose a novel fast and robust 3D point clouds segmentation framework via coupled feature selection, named 3DCFS, that jointly performs semantic and instance segmentation. Inspired by the human scene perception process, we design a novel coupled feature selection module, named CFSM, that adaptively selects and fuses the reciprocal semantic and instance features from two tasks in a coupled manner. To further boost the performance of the instance segmentation task in our 3DCFS, we investigate a loss function that helps the model learn to balance the magnitudes of the output embedding dimensions during training, which makes calculating the Euclidean distance more reliable and enhances the generalizability of the model. Extensive experiments demonstrate that our 3DCFS outperforms state-of-the-art methods on benchmark datasets in terms of accuracy, speed and computational cost. Codes are available at: https://github.com/Biotan/3DCFS.

Despite the great success achieved by supervised fully convolutional models in semantic segmentation, training the models requires a large amount of labor-intensive work to generate pixel-level annotations. Recent works exploit synthetic data to train the model for semantic segmentation, but the domain adaptation between real and synthetic images remains a challenging problem. In this work, we propose a Separated Semantic Feature based domain adaptation network, named SSF-DAN, for semantic segmentation. First, a Semantic-wise Separable Discriminator (SS-D) is designed to independently adapt semantic features across the target and source domains, which addresses the inconsistent adaptation issue in the class-wise adversarial learning. In SS-D, a progressive confidence strategy is included to achieve a more reliable separation. Then, an efficient Class-wise Adversarial loss Reweighting module (CA-R) is introduced to balance the class-wise adversarial learning process, which leads the generator to focus more on poorly adapted classes. The presented framework demonstrates robust performance, superior to state-of-the-art methods on benchmark datasets.