Semantic segmentation of 3D point clouds has garnered considerable attention in academic research and industrial applications. However, existing methods typically assume fixed semantic classes, which is unrealistic for practical scenarios where new classes emerge incrementally. This leads to catastrophic forgetting of previous knowledge and semantic shifts where annotations treat previous classes as background in new tasks. Moreover, the unordered and unstructured properties of point clouds further exacerbate catastrophic forgetting. To address these, we propose the Global-Local Awareness for 3D point cloud class-incremental semantic segmentation (i.e., GLA). Specifically, we introduce a global-aware modeling network (GAMN) based on the state space model, providing robust feature representations. Furthermore, to alleviate catastrophic forgetting, we propose a Graph Attention Knowledge Distillation (GAKD) module. GAKD adaptively integrates local geometric features through an attention mechanism, emphasizing the distillation of geometric structural knowledge. Notably, our approach integrates global context-awareness and local attention through the state space model and GAKD module, respectively. In addition, we introduce pseudo-labeling to alleviate semantic shift. Experiments on the S3DIS dataset validate the superiority of our approach.

Class incremental learning (CIL) has drawn wide attention in academic researches. However, most existing methods cannot be applied to some practical scenarios in which unknown classes occur during the inference stage. To solve this problem, we target a more challenging and realistic setting:Incremental Open Set Learning(IOSL), which needs to reject unknown classes from test data while incrementally learning new classes. IOSL has two coupled key challenges: 1) overcoming the catastrophic forgetting of old classes when learning new classes incrementally due to the rarity of old training samples, and 2) minimizing the empirical classification risk on known classes and the open space risk on unknown classes. To address these challenges, we propose an incremental open-set learning method with a “future-look” ability. This ability reserves embedding space for incrementally arriving new classes and potential unknown classes simultaneously to alleviate the catastrophic forgetting indirectly and recognize unknown classes well. Specifically, a normalized prototype learning strategy is designed to minimize the empirical classification risk and implicitly reserve some space. Moreover, we design an extra classes synthesizing module to explicitly reserve more suitable space. This further minimizes the empirical classification risk while reducing the open space risk. Furthermore, we develop an adaptive metric learning loss to mitigate the class imbalance between old and new classes, which focuses on exploiting exemplars fully and selects an adaptive margin for pairs of old and new classes. Extensive experiments on representative classification datasets validate the superiority of our method.

Three-dimensional (3-D) object recognition has achieved satisfied achievement in both academia and industry. However, most traditional 3-D object classification methods implicitly assume that there are abundant training data from a static distribution. To relax the assumption, we target on a more challenging and realistic setting: few-shot incremental 3-D object learning (FSI3DL), which intends to incrementally classify the new coming 3-D objects with few training data. In order to achieve this, two key challenges need to be concerned: 1) the catastrophic forgetting issue caused by incremental 3-D data with irregular and redundant topological structures and 2) the overfitting issue caused by few-shot training data. To address the first challenge, we use Laplacian spectral analysis based on 3-D meshes to design an embedding network that consists of super-vertex graph convolution (SVGC) module and topology-aware graph attention (TAGA) module. The SVGC is designed to construct the discriminative local topological characteristics for representing the irregular 3-D meshes better. The TAGA is designed to identify redundant topological characteristics. To address the second challenge, a fine-tuning strategy with model alignment regularization is investigated. Furthermore, an embedding space selection and fusion (ESSF) strategy is proposed in the inference phase to mitigate catastrophic forgetting and overfitting further. Combining SVGC, TAGA, and alignment regularization with ESSF strategy, a novel topology-aware graph convolution network (TopGCN) is proposed to address the FSI3DL. Experiments on representative 3-D classification datasets validate the superiority of TopGCN.

3D object recognition has garnered notable success in both academic and industrial contexts. However, the majority of existing 3D object recognition approaches are tailor-made for static scenarios that have ample availability of training instances. Therefore, we target a more arduous and pragmatic task: few-shot incremental 3D object learning (FSI3DL), which aims to learn the new 3D objects in an incremental manner, yet new classes only have a few training instances. However, this task presents two challenges: overfitting to few-shot, and catastrophic forgetting on previous classes due to the absence of previous training instances. To mitigate these challenges, we introduce an Angular Penalty method to consume irregular mesh directly and reserve embedding space for incoming new classes. Specifically, we design a graph convolution network that can take advantage of the better representation ability of mesh and overcome its irregularity. Moreover, we use an angular penalty loss to increase the similarity between inra-class instances and reduce the similarity between instances from different classes. This can leave space in embedding space for incoming new classes. Experiments on representative 3D object classification datasets demonstrate the better efficacy of our method.

Robot manipulation, for example, pick-and-place manipulation, is broadly used for intelligent manufacturing with industrial robots, ocean engineering with underwater robots, service robots, or even healthcare with medical robots. Most traditional robot manipulations adopt 2-D vision systems with plane hypotheses and can only generate 3-DOF (degrees of freedom) pose accordingly. To mimic human intelligence and endow the robot with more flexible working capabilities, 3-D vision-based robot manipulation has been studied. However, this task is still challenging in the open world especially for general object recognition and pose estimation with occlusion in cluttered backgrounds and human-like flexible manipulation. In this article, we propose a comprehensive analysis of recent progress about the 3-D vision for robot manipulation, including 3-D data acquisition and representation, robot-vision calibration, 3-D object detection/recognition, 6-DOF pose estimation, grasping estimation, and motion planning. We then present some public datasets, evaluation criteria, comparisons, and challenges. Finally, the related application domains of robot manipulation are given, and some future directions and open problems are studied as well.

3D object classification has attracted appealing attentions in academic researches and industrial applications. However, most existing methods need to access the training data of past 3D object classes when facing the common real-world scenario: new classes of 3D objects arrive in a sequence. Moreover, the performance of advanced approaches degrades dramatically for past learned classes (i.e., catastrophic forgetting), due to the irregular and redundant geometric structures of 3D point cloud data. To address these challenges, we propose a new Incremental 3D Object Learning (i.e., I3DOL) model, which is the first exploration to learn new classes of 3D object continually. Specifically, an adaptive-geometric centroid module is designed to construct discriminative local geometric structures, which can better characterize the irregular point cloud representation for 3D object. Afterwards, to prevent the catastrophic forgetting brought by redundant geometric information, a geometric-aware attention mechanism is developed to quantify the contributions of local geometric structures, and explore unique 3D geometric characteristics with high contributions for classes incremental learning. Meanwhile, a score fairness compensation strategy is proposed to further alleviate the catastrophic forgetting caused by unbalanced data between past and new classes of 3D object, by compensating biased prediction for new classes in the validation phase. Experiments on 3D representative datasets validate the superiority of our I3DOL framework.

As an important type of geometric data for 3D shapes, the unique topological connection makes the mesh more powerful than other types of data, but it also introduces complexity and irregularity. In this paper, we propose a Topological Perception Network (TPN) that consumes meshes directly to learn 3D shape representation via informative topology property. More specifically, to tackle the complexity and irregularity problem, a Topological Perception Attention (TPA) is designed that could incorporate local topological information efficiently via focusing on more important edges of the local topological neighborhood. Meanwhile, it could be stacked to produce global shape representation. Compared with the state-of-the-art, the proposed TPN uses less than half of the vertex number to get better performance, while costing less memory and computational time. Experiments on ModelNet40 and ShapeNet Core55 datasets demonstrate the effectiveness of our method on classification and retrieval.