In perception, multiple sensory information is integrated to map visual information from 2D views onto 3D objects, which is beneficial for understanding in 3D environments. But in terms of a single 2D view rendered from different angles, only limited partial information can be provided. The richness and value of Multi-view 2D information can provide superior self-supervised signals for 3D objects. In this paper, we propose a novel self-supervised point cloud representation learning method, MM-Point, which is driven by intra-modal and inter-modal similarity objectives. The core of MM-Point lies in the Multi-modal interaction and transmission between 3D objects and multiple 2D views at the same time. In order to more effectively simultaneously perform the consistent cross-modal objective of 2D multi-view information based on contrastive learning, we further propose Multi-MLP and Multi-level Augmentation strategies. Through carefully designed transformation strategies, we further learn Multi-level invariance in 2D Multi-views. MM-Point demonstrates state-of-the-art (SOTA) performance in various downstream tasks. For instance, it achieves a peak accuracy of 92.4% on the synthetic dataset ModelNet40, and a top accuracy of 87.8% on the real-world dataset ScanObjectNN, comparable to fully supervised methods. Additionally, we demonstrate its effectiveness in tasks such as few-shot classification, 3D part segmentation and 3D semantic segmentation.

3D novelty detection plays a crucial role in various real-world applications, especially in safety-critical fields such as autonomous driving and intelligent surveillance systems. However, existing 3D novelty detection methods are constrained by the scarcity of 3D data, which may impede the model's ability to learn adequate representations, thereby impacting detection accuracy. To address this challenge, we propose a Unified Learning Framework (UniL) for facilitating novelty detection. During the pretraining phase, UniL assists the point cloud encoder in learning information from other modalities, aligning visual, textual, and 3D features within the same feature space. Additionally, we introduce a novel Multimodal Supervised Contrastive Loss (MSC Loss) to improve the model's ability to cluster samples from the same category in feature space by leveraging label information during pretraining. Furthermore, we propose a straightforward yet powerful scoring method, Depth Map Error (DME), which assesses the discrepancy between projected depth maps before and after point cloud reconstruction during novelty detection. Extensive experiments conducted on 3DOS have demonstrated the effectiveness of our approach, significantly enhancing the performance of the unsupervised VAE method in 3D novelty detection. Codes are avaliable at https://github.com/EugeneWon9/UniL.

Co-salient object detection (CoSOD) aims to detect common salient objects sharing the same attributes in an image group. The key issue of CoSOD is how to model the inter-saliency relations within an image group. The major limitation of previous methods is that they pre-define the group-to-one relations within an image group. In this paper, we propose a new concept of structural inter-saliency relations and solve the CoSOD with deep reinforcement learning framework. Firstly, we design a semantic relation graph (SRG) to model the structural inter-saliency relations. Then the feature selecting agent (FS-agent) aims to select the informative features, which can help the SRG effectively model structural inter-saliency relations. Finally, relation updating agent (RU-agent) progressively updates the SRG to focus on the co-salient relations like human decision-making process. Extensive experiments on co-saliency datasets show that because of well modeling inter-saliency relations in image group, our proposed method achieves superior performance compared to the state-of-the-art methods. We hope that this paper can motivate future research for visual co-analysis tasks.

In this paper, we present a novel model for simultaneous stable co-saliency detection (CoSOD) and object co-segmentation (CoSEG). To detect co-saliency (segmentation) accurately, the core problem is to well model inter-image relations between an image group. Some methods design sophisticated modules, such as recurrent neural network (RNN), to address this problem. However, order-sensitive problem is the major drawback of RNN, which heavily affects the stability of proposed CoSOD (CoSEG) model. In this paper, inspired by RNN-based model, we first propose a multi-path stable recurrent unit (MSRU), containing dummy orders mechanisms (DOM) and recurrent unit (RU). Our proposed MSRU not only helps CoSOD (CoSEG) model captures robust inter-image relations, but also reduces order-sensitivity, resulting in a more stable inference and training process. Moreover, we design a cross-order contrastive loss (COCL) that can further address order-sensitive problem by pulling close the feature embedding generated from different input orders. We validate our model on five widely used CoSOD datasets (CoCA, CoSOD3k, Cosal2015, iCoseg and MSRC), and three widely used datasets (Internet, iCoseg and PASCAL-VOC) for object co-segmentation, the performance demonstrates the superiority of the proposed approach as compared to the state-of-the-art (SOTA) methods.

Aiming at discovering and locating most distinctive objects from visual scenes, salient object detection (SOD) plays an essential role in various computer vision systems. Coming to the era of high resolution, SOD methods are facing new challenges. The major limitation of previous methods is that they try to identify the salient regions and estimate the accurate objects boundaries simultaneously with a single regression task at low-resolution. This practice ignores the inherent difference between the two difficult problems, resulting in poor detection quality. In this paper, we propose a novel deep learning framework for high-resolution SOD task, which disentangles the task into a low-resolution saliency classification network (LRSCN) and a high-resolution refinement network (HRRN). As a pixel-wise classification task, LRSCN is designed to capture sufficient semantics at low-resolution to identify the definite salient, background and uncertain image regions. HRRN is a regression task, which aims at accurately refining the saliency value of pixels in the uncertain region to preserve a clear object boundary at high-resolution with limited GPU memory. It is worth noting that by introducing uncertainty into the training process, our HRRN can well address the high-resolution refinement task without using any high-resolution training data. Extensive experiments on high-resolution saliency datasets as well as some widely used saliency benchmarks show that the proposed method achieves superior performance compared to the state-of-the-art methods.

Abstract 3D shape recognition has been actively investigated in the field of computer graphics. With the rapid development of deep learning, various deep models have been introduced and achieved remarkable results. Most 3D shape recognition methods are supervised and learn only from the large amount of labeled shapes. However, it is expensive and time consuming to obtain such a large training set. In contrast to these methods, this paper studies a semi‐supervised learning framework to train a deep model for 3D shape recognition by using both labeled and unlabeled shapes. Inspired by the co‐training algorithm, our method iterates between model training and pseudo‐label generation phases. In the model training phase, we train two deep networks based on the point cloud and multi‐view representation simultaneously. In the pseudo‐label generation phase, we generate the pseudo‐labels of the unlabeled shapes using the joint prediction of two networks, which augments the labeled set for the next iteration. To extract more reliable consensus information from multiple representations, we propose an uncertainty‐aware consistency loss function to combine the two networks into a multimodal network. This not only encourages the two networks to give similar predictions on the unlabeled set, but also eliminates the negative influence of the large performance gap between the two networks. Experiments on the benchmark ModelNet40 demonstrate that, with only 10% labeled training data, our approach achieves competitive performance to the results reported by supervised methods.

To efficiently and accurately classify a large 3D shape collection, this paper proposes a novel interactive system by incorporating active learning, online learning and user intervention. Given a shape collection, our system iteratively alternates the interactive annotation and verification until all the shapes are classified. The main advantage is that it provides faster interactive classification rates than alternative approaches. Our system achieves this goal by a unified active learning algorithm that selects the shapes to be annotated or verified, which requires a probability model for simulating the time cost of human input during manual intervention. After manually classifying these selected shapes, we use an extended soft confidence-weighted learning method to update the classifier incrementally and efficiently for the subsequent active selection and shape classification in turn. Experimental results demonstrated the effectiveness of the proposed method.

Co-segmentation of 3D shapes has been receiving increasing attention, and treated as clustering problem in a descriptor space by a few unsupervised approaches to achieve proper co-segmentation of shapes with large variability. However, most of the existing algorithms are performed on segment level and heavily dependent on the per-object segmentation. Accordingly, we propose a co-segmentation method based on combination of Fuzzy C-Means (FCM) and Random Walks together. The novelty of our method is twofold. As an efficient soft clustering algorithm, FCM is firstly used to cluster directly all the facets in the set in terms of their shape descriptors. The clusters of facets are created as candidates of the consistent parts of shapes. Random Walks model is then incorporated into the iterations of FCM clustering to adjust the assignment of facets in each candidate according to the minima rule of shape segmentation. The results of co-segmentation are refined through the iterations of FCM until its convergence conditions are satisfied. Experiments prove that the method proposed in this paper can not only get more stable results without per-object segmentation, but also improve the accuracy of co-segmentation.

In this poster, an approach for best view selection of 3D models is proposed, which is based on the framework that formulates the selection as a problem of evaluating views' discrimination ability. Firstly, different views' features are extracted by unsupervised feature learning. Then classifiers are trained to evaluate each view's discrimination ability. A view with the best classifier has the best discrimination ability, and it is chosen as the best view of the 3D model. At last, experiments show that 3D models of same class have similar best views.

This paper presents an intent-driven model synthesis method. The method introduces an interactive straight prismatic construction space to realize the structure and shape variation of the example model simultaneously. The construction space defines the global size and the local shape feature of the desired model. Users can draw a closed curve and some skeleton lines by a sketch-based interface to design the construction space. Our algorithm first uses a quadrangulation algorithm to create a subdivision plane with the same contour as the closed curve. And the drawn skeleton lines control the local orientation of split units in the plane. Then it creates the construction space by sweeping the subdivision plane. Finally, it fills the construction space with the deformed model pieces while maintaining the generalized adjacent constraints, which are defined according to the example model. We demonstrate the effectiveness of the approach on large-scale complex models such as architecture, mountains.

This paper presents a synthesis method for 3D models using Petri net. Feature structure units from the example model are extracted, along with their constraints, through structure analysis, to create a new model using an inference method based on Petri net. Our method has two main advantages: first, 3D model pieces are delineated as the feature structure units and Petri net is used to record their shape features and their constraints in order to outline the model, including extending and deforming operations; second, a construction space generating algorithm is presented to convert the curve drawn by the user into local shape controlling parameters, and the free form deformation (FFD) algorithm is used in the inference process to deform the feature structure units. Experimental results showed that the proposed method can create large-scale complex scenes or models and allow users to effectively control the model result.

We propose a 3D model feature line extraction method using templates for guidance. The 3D model is first projected into a depth map, and a set of candidate feature points are extracted. Then, a conditional random fields (CRF) model is established to match the sketch points and the candidate feature points. Using sketch strokes, the candidate feature points can then be connected to obtain the feature lines, and using a CRF-matching model, the 2D image shape similarity features and 3D model geometric features can be effectively integrated. Finally, a relational metric based on shape and topological similarity is proposed to evaluate the matching results, and an iterative matching process is applied to obtain the globally optimized model feature lines. Experimental results showed that the proposed method can extract sound 3D model feature lines which correspond to the initial sketch template.