3D full-scene segmentation technology has demonstrated great potential driven by large models, but it often faces challenges of incomplete scenes and identification of invisible classes in practical applications. To address this, we propose the LR-AdaInSeg method, which significantly enhances the model’s generalization ability in incomplete scenes through two key innovations: First, we design a Bayesian Low-Rank Module, which effectively solves the problem of feature space redundancy through dynamic optimization of the network structure, improving adaptability to incomplete scenes. Second, we combine graph contrastive clustering with the Low-Rank module, leveraging its robust feature representation capability to achieve accurate differentiation of invisible classes. In terms of implementation, we build a multi-scale feature extraction framework based on the 3D U-Net and utilize the 3D prompt points and their 2D masks as supervisory signals to achieve effective fusion of geometric and semantic information. Experiments show that our method achieves advanced performance on multiple benchmarks such as ScanNet, particularly excelling in handling incomplete scenes and invisible class objects.

Open-world semi-supervised learning (OWSSL) extends traditional semi-supervised learning to open-world scenarios by identifying novel categories in unlabeled data, thereby enhancing the model's generalization capability. However, existing OWSSL datasets typically assume a balanced class distribution, whereas real-world applications often exhibit highly imbalanced distributions. This imbalance makes it particularly challenging to learn tail classes and discover novel categories. This paper introduces the Class-Balanced Representation and Recognition Framework (CBTM-NCD), which uses the Variational Dirichlet Process (VDP) to improve tail class features and includes a generative data balancing strategy.Additionally, CBTM-NCD adopts a two-stage optimization strategy to identify novel category samples, effectively tackling three major challenges prevalent in open-world long-tailed scenarios in open-world long-tailed distributions: insufficient feature representation of tail classes, difficulty in discovering unknown categories, and class distribution imbalance.To enhance transparency and reproducibility, the code is available at https://github.com/wuzelei123/CBTM-NCD.

Spectral clustering is widely used in the field of unsupervised learning and has been successfully applied in various data analysis tasks. However, in practical scenarios, it’s common to get few labeled data at low cost, resulting limited labeled data in few categories, while the rest are completely unlabeled. Relying solely on the unsupervised clustering framework can’t leverage the existing supervised information. Using a supervised meta-learning paradigm alone poses challenges when dealing with the majority categories of completely unlabeled data. Therefore, we propose a few-shot contrastive spectral clustering combined with meta learning and neighbor mining framework (FCSMN). First we leverage few shot labeled information to obtain a meta-learning feature embedding with strong generalization capabilities, and acquire semantic feature spaces using an unsupervised contrastive learning model. Then, we mine neighbors in these two feature spaces separately and align them. Next, we jointly reinforce intra-cluster compactness and inter-cluster separability at both instance and cluster level. To satisfies the conditions of spectral clustering, we impose orthogonality constraints at the last layer. Experiments on three datasets demonstrate the effectiveness our proposed method.

Abstract In the field of object detection, there is often a high level of occlusion in real scenes, which can very easily interfere with the accuracy of the detector. Currently, most detectors use a convolutional neural network (CNN) as a backbone network, but the robustness of CNNs for detection under cover is poor, and the absence of object pixels makes conventional convolution ineffective in extracting features, leading to a decrease in detection accuracy. To address these two problems, we propose VFN (A Vision Enhancement and Feature Fusion Multiscale Detection Network), which first builds a multiscale backbone network using different stages of the Swin Transformer, and then utilizes a vision enhancement module using dilated convolution to enhance the vision of feature points at different scales and address the problem of missing pixels. Finally, the feature guidance module enables features at each scale to be enhanced by fusing with each other. The total accuracy demonstrated by VFN on both the PASCAL VOC dataset and the CrowdHuman dataset is better than that of other methods, and its ability to find occluded objects is also better, demonstrating the effectiveness of our method.The code is available at https://github.com/qcw666/vfn .

Abstract Deep spectral clustering techniques are considered one of the most efficient clustering algorithms in data mining field. The similarity between instances and the disparity among classes are two critical factors in clustering fields. However, most current deep spectral clustering approaches do not sufficiently take them both into consideration. To tackle the above issue, we propose Semantic Spectral clustering with Contrastive learning and Neighbor mining (SSCN) framework, which performs instance-level pulling and cluster-level pushing cooperatively. Specifically, we obtain the semantic feature embedding using an unsupervised contrastive learning model. Next, we obtain the nearest neighbors partially and globally, and the neighbors along with data augmentation information enhance their effectiveness collaboratively on the instance level as well as the cluster level. The spectral constraint is applied by orthogonal layers to satisfy conventional spectral clustering. Extensive experiments demonstrate the superiority of our proposed frame of spectral clustering.

As one of the most effective subspace clustering methods, the self-expression based sparsity method leverages the robust representational learning and non-linear transformation capacities of deep learning. This approach facilitates the mapping of data into a low-dimensional subspace, wherein the clustering operations are subsequently executed. However, most conventional self-expression methods do not handle the subspace clustering problem with sparse-labeled information. Considering the scarcity and value of labeled samples in various real-world applications, we propose a novel deep Few-Shot Subspace Clustering Learning (FS2CL) framework to improve the traditional self-expression-based techniques in the case of sparse label information, in which partial classes in the observation dataset have a scarcity of labeled samples and most other classes do not. We expect to obtain more discriminative low-rank representations that exhibit high cohesion among clusters. To overcome the limitation that the low-rank approximation is achieved by singular value decomposition, which is not differentiable and cannot be embedded in neural networks for gradient backpropagation, a Low-rank Representation Approximation (LRA) module is proposed to transform the non-differentiable singular value decomposition into a differentiable iterative process. This procedure produces a low-rank representation that maximizes the cohesion of features belonging to the same cluster. Subsequently, we propose a method for learning a low-dimensional learnable subspace bases matrix assisted by a small number of labeled samples, which captures the structure of each subspace. We then classify the data points belonging to the corresponding class by measuring the similarity between the instance and each subspace base. Due to the low dimension of the subspace bases matrix, it is possible to apply our method to large-scale datasets. The proposed method is superior to state-of-the-art clustering approaches through extensive comparison studies conducted on six benchmark datasets: MNIST, Fashion-MNIST, REUTERS-10K, STL-10, CIFAR10, and CIFAR100.

Abstract Portrait‐background image composition is a widely used operation in selfie editing, video meeting, and other portrait applications. To guarantee the realism of the composited images, the appearance of the foreground portraits needs to be adjusted to fit the new background images. Existing image harmonization approaches are proposed to handle general foreground objects, thus lack the special ability to adjust portrait foregrounds. In this paper, we present a novel end‐to‐end network architecture to learn both the content features and style features for portrait‐background composition. The method adjusts the appearance of portraits to make them compatible with backgrounds, while the generation of the composited images satisfies the prior of a style‐based generator. We also propose a pipeline to generate high‐quality and high‐variety synthesized image datasets for training and evaluation. The proposed method outperforms other state‐of‐the‐art methods both on the synthesized dataset and the real composited images and shows robust performance in video applications.

The graph convolutional network (GCN), as a powerful tool in graph data processing, is widely exploited in many machine learning and computer vision tasks. However, existing GCNs usually assume that the network has fixed outputs, which is usually contrary to the real-world class number being unknown and incremental, leading to an open set classification problem in which the finite training dataset cannot contain all labels in the infinite testing data. To overcome these issues, a novel Bayesian model is proposed, in which we couple GCN and a deep generative clustering model in a unified framework. In our model, the GCN model is used to detect the known classes, the deep generative clustering model is designed to generate the novel classes, and a two-level label generative process is constructed to extend the finite GCN outputs to infinity and fuse the label generated by the GCN model and the deep generative model. Although posterior inference is difficult, our model leads to an efficient variational inference-based optimization method. Experiments on various datasets validate our theoretical analysis and demonstrate that our model can achieve state-of-the-art performance. Our source code has been released on the website.

Deep spectral clustering is a popular and efficient algorithm in unsupervised learning. However, deep spectral clustering methods are organized into three separate steps: affinity matrix learning, spectral embedding learning, and K-means clustering on spectral embedding. In this case, although each step can achieve its own performance, it is still difficult to obtain robust clustering results. In this letter, we propose a one-step adaptive spectral clustering network to overcome the aforementioned shortcomings. The network embeds the three parts of affinity matrix learning, spectral embedding learning, and indicator learning into a unified framework. The affinity matrix is adaptively adjusted by spectral embedding in a deep subspace. We introduce spectral rotation to discretize spectral embedding, which makes the spectral embedding and indicator be learned simultaneously to improve clustering quality. Each part of the model can be iteratively updated based on other parts to optimize the clustering results. Experimental results on four real datasets show the effectiveness of our method on the ACC and NMI clustering evaluation metrics. In particular, our method achieves an NMI of 0.932 and an ACC of 0.973 on the MNIST dataset, a decent performance boost compared to the best baseline.

Sparse subspace clustering, as one of the most effective subspace clustering method, is widely studied in the data processing realm. However, conventional sparse subspace clustering methods are organized with two separated steps, feature learning and indicator learning. This makes the algorithm suffer the difficulties that: (1) representation and cluster indicator learning cannot affect each other; (2) cluster number and sparse penalty coefficient should be specified a priori; (3) sparse subspace clustering method is designed for the linear subspace data and cannot be exploited in the general clustering task. In this paper, a novel sparse clustering method is proposed, in which we extend the conventional algebraic sparse subspace clustering approach to a Bayesian framework. Then, cluster number estimation and low rank constraint are coupled via a Dirichlet process parameter generation process, in which the rank are no more required to be low but can be generated with a suitable value. Finally, Generative Adversarial Network (GAN) is incorporated into the Bayesian sparse model, which extends the subspace clustering method to a normal clustering model. Experiments on different real world datasets validate our theory analysis and demonstrate the effective of the proposed algorithm.

The principle of generative adversarial net is to fit the given data distribution by combining a generative model and discriminative model. There are two major challenges to conventional systems – they are difficult to train and they easily fall into ‘mode collapse’. To improve it, this study describes a novel network structure with dual generators. A ‘cooperation’ mechanism is introduced to help the generators work together. During training, generators not only learn from discriminative feedback but also from each other (like a study group). Compared with a single‐generator network, a dual‐generator network could capture many more ‘modes’ and eventually reduce the impact of ‘mode collapse.’ Dual networks also require extra computational resources. However, our experiment shows that even with network parameters of similar size, dual networks still achieved better results. Additionally, a dual‐generator structure could be extended to multiple generators. The proposed network structure is also very robust and flexible. It can be adapted to various application scenarios, such as high‐resolution image generation, domain adaptation and 3D model generation. The experimental results showed that with the same computing resources, multiple generators can generate better quality synthetic data, including 2D images, 3D objects, style transferring etc.

Spectral clustering is a popular tool in many unsupervised computer vision and machine learning tasks. Recently, due to the encouraging performance of deep neural networks, many conventional spectral clustering methods have been extended to the deep framework. Although these deep spectral clustering methods are quite powerful and effective, learning the cluster number from data is still a challenge. In this paper, we aim to tackle this problem by integrating the spectral clustering, generative adversarial network and low rank model within a unified Bayesian framework. First, we adapt the low rank method to the cluster number estimation problem. Then, an adversarial-learning-based deep clustering method is proposed and incorporated. When introducing the spectral clustering method into our model clustering procedure, a hidden space structure preservation term is proposed. Via a Bayesian framework, the structure preservation term is embedded into the generative process, which can then be used to deduce a spectral clustering in the optimization procedure. Finally, we derive a variational-inference-based method and embed it into the network optimization and learning procedure. Experiments on different datasets prove that our model has the cluster number estimation capability and show that our method can outperform many similar graph clustering methods.

Being a widely studied model in machine learning and multimedia community, Deep Neural Network (DNN) has achieved an encouraging success in various applications. However, conventional DNN suffers the difficulty when handling the open set learning problem, in which the true class number is unknown, and the predication label in the testing dataset usually has unseen classes which are not contained in the training set. In this paper, we aim to tackle this problem by unifying deep neural network and Dirichlet process mixture model. Firstly, to learn the deep feature and enable the incorporation of DNN and the Bayesian nonparametric model, we extend deep metric learning to a semi-supervised framework. Secondly, with the learned deep feature, we construct our open set classification method by expanding the Dirichlet process mixture model to a semi-supervised framework. To infer our semi-supervised Bayesian model, the corresponding variational inference algorithm has also been derived. Experiment on synthetic and real world datasets validates our theory analysis and demonstrates the state-of-the-art performance.

Multimanifold clustering separates data points approximately lying on a union of submanifolds into several clusters. In this paper, we propose a new nonparametric Bayesian model to handle the manifold data structure. In our framework, we first model the manifold mapping function between Euclidean space and topological space by applying a deep neural network, and then construct the corresponding generation process of multiple manifold data. To solve the posterior approximation problem, in the optimization procedure, we apply a variational auto-encoder-based optimization algorithm. Especially, as the manifold algorithm has poor performance on the real dataset where nonmanifold and manifold clusters are appearing simultaneously, we expand our proposed manifold algorithm by integrating it with the original Dirichlet process mixture model. Experimental results have been carried out to demonstrate the state-of-the-art clustering performance.