Protein-Ligand Affinity (PLA) prediction quantifies the interaction strength to guide rational drug design. Existing approaches typically analyze interaction at a single granularity and overlook tightly coupled relationships between protein and ligand in both structure and functionality, consequently yielding suboptimal representations, leading to significant performance drops in real-world scenarios. To address this problem, we propose PLA-MGRA, a minimalist and effective PLA prediction framework. Specifically, PLA-MGRA captures both fine-grained atomic details and coarse grained functional semantics within the 3D structure of protein–ligand complexes, through multi-granularity learning. To further parse the coupled protein–ligand relationships, we design relation-aware learning to enhance the binding nature of representations. Extensive experiments demonstrate that our method achieves state-of-the-art performance on multiple protein–ligand affinity prediction benchmarks, while also offering generalizability and interpretability.

Spatial transcriptomics enables gene expression profiling with spatial context, offering unprecedented insights into the tissue microenvironment. However, most computational models treat genes as isolated numerical features, ignoring the rich biological semantics encoded in their symbols. This prevents a truly deep understanding of critical biological characteristics. To overcome this limitation, we present SemST, a semantic-guided deep learning framework for spatial transcriptomics data clustering. SemST leverages Large Language Models (LLMs) to enable genes to "speak" through their symbolic meanings, transforming gene sets within each tissue spot into biologically informed embeddings. These embeddings are then fused with the spatial neighborhood relationships captured by Graph Neural Networks (GNNs), achieving a coherent integration of biological function and spatial structure. We further introduce the Fine-grained Semantic Modulation (FSM) module to optimally exploit these biological priors. The FSM module learns spot-specific affine transformations that empower the semantic embeddings to perform an element-wise calibration of the spatial features, thus dynamically injecting high-order biological knowledge into the spatial context. Extensive experiments on public spatial transcriptomics datasets show that SemST achieves state-of-the-art clustering performance. Crucially, the FSM module exhibits plug-and-play versatility, consistently improving the performance when integrated into other baseline methods.

Despite the success of convolution- and attention-based models in vision tasks, their rigid receptive fields and complex architectures limit their ability to model irregular spatial patterns and hinder interpretability, thereby posing challenges for tasks requiring high model transparency. Clustering paradigms offer promising interpretability and flexible semantic modeling, but suffer from limited accuracy, low efficiency, and gradient vanishing during training. To address these issues, we propose the CLUster attEntion Network (CLUENet), a transparent deep architecture for visual semantic understanding. Specifically, we introduce three key innovations, including (i) a Global and Soft Feature Aggregation with a Temperature-Scaled Cosine Attention for capturing long-range dependencies and a Gated Fusion Mechanism for enhanced local modeling, (ii) Hard and Shared Feature Dispatching, and (iii) an Improved Cluster Pooling Block. These enhancements significantly improve both classification performance and visual interpretability. Experiments on CIFAR-100 and Mini-ImageNet demonstrate that CLUENet outperforms existing clustering methods and mainstream visual models, offering a compelling balance of accuracy, efficiency, and transparency.

The task of image feature matching aims to establish correct correspondences between images from two different views. While approaches based on attention mechanisms have demonstrated remarkable advancements in image feature matching, they still encounter substantial limitations. Specifically, current graph attention network approaches face performance bottlenecks in complex scenarios, such as low-texture regions or occlusions. This limitation stems from the self-attention mechanism, which, when lacking effective guidance, can lead to divergent attention weights or incorrect focus on regions with low discriminability, resulting in matching failures in low-texture environments. Inspired by how humans focus on distinctive regions when performing cross-view matching, we enhance attention to singular points in images that are salient, unique and have high cross-view matching potential during information aggregation, thereby improving matching capability. To realize the aforementioned strategies, we develop a novel Singularity-enhanced Graph Attention Network (SGAT). SGAT leverages Co-potentiality and Multi-Scale Singularity as prior guidance, and designs a Singularity-aware Attention mechanism and a Co-potentiality Guided Attention mechanism , specifically enhancing the perception of singularity and matching potential during feature interaction. Experimental results on multiple datasets, including ScanNet1500, demonstrate that our method outperforms current state-of-the-art sparse matching methods. In particular, the improvement is most pronounced in complex scenarios such as low-texture environments, significantly enhancing the accuracy and robustness of image matching and its downstream tasks.

Multimodal Emotion Recognition (MER) aims to understand complex human emotions by jointly analyzing visual and textual data. However, in real-world scenarios, emotional cues from different modalities often contain conflict information, such as a smiling face paired with negative text, which poses great challenges for existing multimodal language models (MLLMs). Existing emotion MLLMs and multimodal emotion benchmarks often overlook or even intentionally avoid scenarios involving multimodal emotion conflicts, limiting their ability to reason about complex and contradictory affective cues. By addressing this, we propose Conflict-Explicit Reflective Agent (CERA), a training-free, conflict-aware, and language-driven agentic framework for MER. The concept of CERA is to treat modality emotion conflicts as meaningful signals and resolve them via a three-stage perception–evaluation–reflection reasoning loop. Firstly, the agent’s conflict-perceptive emotion graph construction module builds emotion graphs from fine-grained cues to reveal conflicts, and progressively refines them through iterative updates. Secondly, a reward model evaluates these graphs and produces natural language feedback that identifies unresolved conflicts. Lastly, the language-driven conflict refinement module generates graph editing signals from the feedback without any parameter tuning, enabling the overall CERA to refine its reasoning without training. Extensive experiments on two multimodal emotion datasets, MAFW and CH-SIMS, demonstrate that CERA significantly outperforms state-of-the-art training-free methods in both recognition accuracy and conflict interpretability, providing an effective training-free solution for complex emotional reasoning.

Trustworthy multi-modal learning reliably integrates multiple data sources. However, current methods often face a challenge, which is the inherent non-convex nature of deep neural networks. It leads to their susceptibility to local minima, ultimately resulting in a reduced capacity for generalization. To address this issue, we first create a theoretical framework, which extends the application of curriculum learning in multi-modal scenarios. Secondly, we propose a novel curriculum termed the Dynamic SRM Curriculum (DSRMC). It consists of two modules: a scoring function and a training schedule. The scoring function sorts samples from simple to complex. The training scheduler aims to manage the quantity of samples supplied at each round during training. DSRMC facilitates positioning the learned model in a flatter region of the loss landscape, thereby enhancing its overall generalization ability. Building on DSRMC, we eventually propose an innovative method termed as Curriculum Trustworthy Multi-modal Learning (CTML). It applies DSRMC in multi-modal learning application scenarios. Extensive experiments conducted on three open datasets show that the proposed CTML outperforms state-of-the-art methods, with a maximum improvement of 6.7% in macro F1 score. Our code and dataset are publicly available on https://github.com/HackerHyper/DSRMC.git .

Multi-view clustering (MvC) aims to integrate information from different views to enhance the capability of the model in capturing the underlying data structures. The widely used joint training paradigm in MvC potentially does not fully leverage the multi-view information, due to the imbalanced and under-optimized view-specific features caused by the uniform learning objective for all views. For instance, particular views with more discriminative information could dominate the learning process in the joint training paradigm, leading to other views being under-optimized. To alleviate this issue, we first analyze the imbalanced phenomenon in the joint-training paradigm of multi-view clustering from the perspective of gradient descent for each view-specific feature extractor. Then, we propose a novel balanced multi-view clustering (BMvC) method, which introduces a view-specific contrastive regularization (VCR) to modulate the optimization of each view. Concretely, VCR preserves the sample similarities captured from the joint features and view-specific ones into the clustering distributions corresponding to view-specific features to enhance the learning process of view-specific feature extractors. Additionally, an analysis is provided to illustrate that VCR adaptively modulates the magnitudes of gradients for updating the parameters of view-specific feature extractors to achieve a balanced multi-view learning procedure. In such a manner, BMvC achieves a better trade-off between the exploitation of view-specific patterns and the exploration of view-invariance patterns to fully learn the multi-view information for the clustering task. Finally, a set of experiments are conducted to verify the superiority of the proposed method compared with state-of-the-art approaches both on eight benchmark MvC datasets and two spatially resolved transcriptomics datasets.

Multi-View Clustering (MVC) has gained significant attention for its ability to leverage complementary information across diverse views. However, existing deep MVC methods often struggle with view-distribution entanglement during cross-view fusion, which hampers the quality of the shared latent space and leads to suboptimal clustering performance. To address this issue, we propose the Generalized Multi-view Auto-Encoder (GMAE), a framework designed to preserve cross-view complementarity through disentangled representation learning. Specifically, GMAE employs dual-path autoencoders to decouple source features into view-specific and view-common embeddings, facilitating the discovery of clearer clustering structures. We further construct cross-view adversarial discriminators to guide view-specific encoders in capturing more discriminative features. By strategically modulating mutual information, GMAE effectively aligns distributions and prevents representation collapse, ensuring the generation of robust, non-trivial embeddings. Comprehensive experiments on 13 benchmark datasets demonstrate that GMAE consistently outperforms state-of-the-art methods in both complete and incomplete MVC tasks.

In recent years, Multi-View Clustering (MVC) has been significantly advanced under the influence of deep learning. By integrating heterogeneous data from multiple views, MVC enhances clustering analysis, making multi-view fusion critical to clustering performance. However, multi-view fusion remains challenged by low-quality data, primarily stemming from two reasons: 1) Certain views are contaminated by noisy data. 2) Some views suffer from missing data. This paper proposes a novel Stochastic Generative Diffusion Fusion (SGDF) method to address this problem. SGDF leverages a multiple generative mechanism for the multi-view feature of each sample. It exhibits robustness against low-quality data. Building on SGDF, we further present the Generative Diffusion Contrastive Network (GDCN). Extensive experiments show that GDCN achieves the state-of-the-art results in deep MVC tasks. The source code is publicly available athttps://github.com/HackerHyper/GDCN.

Recent advancements in 3D Gaussian Splatting have achieved high-quality and real-time novel view synthesis for 3D scenes. However, this method primarily focuses on appearance and geometric modeling, lacks the ability to comprehend scenes with fine granularity at the object level. Some techniques enhance Gaussian Splatting, empowering it with the capability to perform unified 3D reconstruction and segmentation on real-world 3D scenes. Despite its improvements, these methods still face challenges in segmentation accuracy and reconstruction quality using 3D Gaussian representation. Additionally, the use of 16-bit identity codes to distinguish different Gaussian groups significantly increases memory overhead. To address these issues, we propose Identity-Encoding Half-Gaussian (ID-HGS) kernels. Our approach introduces a plane to split each Gaussian into two parts with distinct opacity values, enabling precise reconstruction of details and object boundaries. We replace the adaptive density control (ADC) used in Gaussian Grouping with localized half—gaussian point management (LHPM), which performs finer densification in under-reconstructed regions. LHPM resets pathological Gaussians and optimizes Gaussian density, reducing their impact on segmentation accuracy. Furthermore, we assign a global contribution score to each Gaussian and prune low-contribution Gaussians during training, saving memory and accelerating training. Compared to Gaussian Grouping, our method improves both reconstruction quality and segmentation accuracy while effectively controlling memory usage. Extensive experiments demonstrate that HGS-3D outperforms prior Gaussian Grouping on both reconstruction and segmentation: it achieves higher mask accuracy on the LERF-Localization benchmark and reduces peak memory usage while improving render quality on the mipnerf360 dataset.

The anchor-based multi-view clustering method has recently attracted considerable attention due to its superior efficiency. However, most existing methods construct a consensus anchor graph based solely on view-level contributions, overlooking the varying importance of individual samples across different views. Moreover, these methods fail to ensure that anchors are evenly distributed across clusters. Thus, we propose a novel and scalable multi-view clustering method, called Sample-Level Weighted and Structure-Enhanced Anchor Graph Learning for Scalable Multi-View Clustering (SLWSE-AGL). Specifically, we introduce a sample-level weighting mechanism based on anchor self-representation learning, enabling the constructed consensus anchor graph to capture the varying importance of samples in different views. Additionally, we incorporate a structure-enhancement constraint to encourage the learned anchors to be more evenly distributed among clusters, leading to more balanced and meaningful cluster partitions. Furthermore, we employ an anchor-to-sample label propagation mechanism that directly yields the final clustering results, thereby avoiding the information loss associated with the two-stage clustering processes. Extensive experiments demonstrate the superior performance of our method compared to state-of-the-art multi-view clustering approaches. The code of SLWSE-AGL is publicly available at https://github.com/tangchuan2000/SLWSE-AGL.

Multi-view clustering, which uses information from multiple views to partition data into distinct clusters, has garnered significant attention. Existing MLP-based and GCN-based algorithms primarily focus on enhancing performance by extracting node attribute and graph structure features, and then integrating them for clustering. However, features extracted by MLP and GCN may differ in quality due to the high sparsity and noise in multi-view data. Directly integrating these features can lead to feature contamination. To address this issue, we propose a novel mutual calibration network for multi-view clustering (McMVC). Specifically, the attribute and structural features from multi-view data are integrated separately using the attention fusion module. A classifier is then employed to obtain high-confidence pseudo-labels for these fused features. We design a Cluster-level Mutual Calibration (CLMC) module that uses the pseudo-labels to mutually calibrate view features while maintaining compact cluster structures. During training, the attribute and structural features are not directly integrated to avoid feature contamination but are jointly optimized by reliable class information. Concurrently, we construct a Centroid-level Contrastive Calibration (CLCC) module to map view features into their inner centroid space and learn more discriminative centroid representations. Our approach outperforms state-of-the-art methods in multi-view data clustering, as demonstrated by extensive experiments on six real-world benchmark datasets. The source code is available at https://github.com/YuangXiao/ McMVC.

Tensor-based multi-view clustering has been widely studied to capture high-order correlations among multiple views. Nevertheless, existing tensorial methods still exhibit several limitations. First, many approaches rely on full similarity graphs, leading to quadratic or cubic complexity in the number of samples and poor scalability. Second, view-specific anchor graphs are often tensorized without cross-view anchor alignment, yielding structurally inconsistent tensor representations and reduced cross-view comparability. Third, low-rank regularization is typically imposed via the tensor nuclear norm (TNN), which uniformly shrinks singular values and may bias the estimation of the intrinsic tensor rank. To this end, we propose a novel framework, named Align then Tensorize: Multi-level Consistent Anchor Graph Learning for Scalable Multi-View Clustering (ATTMVC). It adopts an anchor-based graph learning framework in which each view is reconstructed from a small set of anchors with sample-wise sparse noise, substantially reducing computational complexity. Unlike existing tensor-based methods that directly tensorize unaligned view-wise anchor graphs, ATTMVC first aligns view-specific anchor graphs into a shared latent space, thereby enforcing structural consistency across views and enabling more reliable modeling of cross-view higher-order correlations. Furthermore, we introduce a Threshold Tensor Rank (TTR) surrogate on the aligned anchor graph tensor, which effectively promotes low-rank structure while mitigating the over-shrinking effect commonly caused by TNN-based regularization. Finally, extensive experiments demonstrate that ATTMVC outperforms state-of-the-art multi-view clustering methods. The code is publicly available at https://github.com/tangchuan2000/ATTMVC.

Accurate identification and classification of white blood cells are essential for diagnosing hematological malignancies and analyzing blood disorders. Existing approaches predominantly leverage masked autoencoders (MAEs) to extract intrinsic blood cell features through image reconstruction as a pretext task. However, these methods encounter two critical challenges: (1) their generalization performance deteriorates under domain shifts caused by variations in staining techniques, illumination conditions, and microscope settings, and (2) the learned data distribution often deviates from the true distribution of blood cell features. To overcome these limitations, we propose CD-CBC, a novel framework for cross-domain blood cell image classification that integrates contrastive representation learning with a denoising mechanism. CD-CBC consists of two key components: a LoRA-based segmentation anything model (LoRA-SAM) and a contrastive masked autoencoder (CMAE). LoRA-SAM mitigates shortcut learning in contrastive learning by eliminating background noise and platelet interference, while CMAE captures fine-grained semantic features and models spatial relationships, enhancing cross-domain robustness. Additionally, we introduce a denoising mechanism in the latent space, which guides the model to focus on unmasked patches during reconstruction, allowing it to better capture the true distribution of blood cell features. Extensive experiments on two benchmark blood cell datasets demonstrate that CD-CBC achieves superior cross-domain performance, reaching an average accuracy of 62.47%, which is 3.17% higher than the current state-of-the-art, thereby confirming its strong generalization capability.

Early detection of colorectal polyps is crucial for clinical screening and cancer prevention, where accurate and efficient automatic segmentation plays a pivotal role. However, colonoscopy images often suffer from low contrast, blurred boundaries, and scale variations, making segmentation challenging. Existing encoder-decoder networks (e.g., U-Net) suffer from asymmetric supervision and feature redundancy, which in turn lead to semantic inconsistency and loss of fine details. While deeper or hybrid designs alleviate these issues, their high complexity and computational burden limit feasibility in real-time clinical practice. To address these challenges, we propose a lightweight segmentation framework, Hierarchical Contextual Attention Network (HCA-Net), consisting of the Redundancy-Suppressed Dual-Path Downsampling (RS-DPD) module and the Boundary-Aware Semantic Alignment Upsampling (BA-SAU) module, applied to the encoder and decoder, respectively. RS-DPD suppresses redundancy while preserving fine-grained details through a dual-path design, whereas BA-SAU leverages cross-layer contextual attention to enforce semantic consistency and enhance boundary sensitivity. Both modules are built upon our proposed Hierarchical Contextual Attention (HCA) mechanism, which combines convolutional projection with pooling-based compression to achieve efficient global modeling and accurate local boundary restoration. In addition, a composite boundary-aware loss function is designed to improve pixel-level accuracy, structural consistency, and robustness in low-contrast and boundary-ambiguous regions. Extensive experiments on public colorectal polyp datasets demonstrate that HCA-Net achieves state-of-the-art (SOTA) segmentation accuracy with significantly improved efficiency, while maintaining robustness under low-contrast and blurred-boundary conditions.

Multimodal clustering (MMC) overcomes the limitations of unimodal methods by integrating information from multiple sources, but the complexity of heterogeneous information coupling hinders effective feature extraction. Critically, existing MMC paradigms primarily focus on capturing consensus through coarse-grained cross-modal alignment. However, such task-agnostic strategies overlook the differences in the utility of feature information across varying task environments. In the absence of task-centric guidance, models often struggle to effectively distinguish task-relevant critical information from task-irrelevant redundant noise during the disentanglement process, leading to information confusion in the representation space. To address this challenge, we propose a deep disentangled multimodal clustering method guided by information theory, named DRLMMC, which employs a tripartite information optimization mechanism to achieve deep disentanglement of cross-modal representations. 1) We design modality-specific encoders to construct nonlinear mapping spaces, transforming the reconstruction mechanism of autoencoders into an information-theoretic mutual information (MI) constraint problem, preserving the unique features of different modalities; 2) To establish cross-modal semantic associations, it constructs a cross-modal shared information extraction module, and, based on an information-theoretic framework, designs an optimization objective function to progressively align multimodal feature subspaces through MI maximization and contrastive learning, capturing task-relevant invariant features across modalities; 3) A unique information dynamic perception module is proposed, which employs a conditional MI projection network combined with learning distribution regularization to adaptively extract and enhance modality-specific task-relevant unique information. Experimental results demonstrate that DRLMMC outperforms existing state-of-the-art methods on multimodal benchmark datasets, exhibiting excellent generalization ability. Notably, it achieves precise disentanglement of cross-omics features in multi-omics analysis, offering a novel methodological approach for handling complex biomedical data.

Anchor graph learning has become a widely used technique for significantly reducing the computational complexity in existing multi-view clustering methods. However, most existing approaches select anchors independently for each view and then generate the consensus graph by directly fusing all anchor graphs. This process overlooks the correspondence between anchor sets across different views, i.e., the column order correspondence of the anchor graphs. To address this limitation, we propose a novel anchor-based tensor multi-rank constraint multi-view clustering method (TMC). Specifically, TMC captures the high-order structural information of the original data by constructing an anchor graph tensor and enforcing a multi-rank constraint to induce a block-diagonal structure. Additionally, to enhance anchor consistency across all view, we construct the anchor graph of each view into an anchor tensor and impose a low-rank constraint on it. In this way, the block-diagonal structure of each anchor graph maintains an approximate alignment between anchors. Furthermore, we provide theoretical proof that the generated anchor graphs inherently exhibit a block-diagonal structure. Extensive experimental results on six multi-view datasets demonstrate that TMC outperforms existing state-of-the-art methods, highlighting its effectiveness in multi-view clustering task.

Multi-View Clustering (MVC) has gained increasing attention due to its ability to effectively leverage the complementary information of multi-view data. Despite the success of existing MVC methods in many real-world applications, they often overlook the discrepancy of view-specific latent distribution and struggle to ensure the completeness of the multi-view data. To address these challenges and harness the powerful feature extraction capability of deep networks, we propose a novel Contrastive and Dual Adversarial Representation Learning method for Multi-view Clustering, termed as CDARL, to solve multi-view clustering problems with both complete and incomplete multi-view data. Specifically, CDARL employs alternating adversarial and contrastive learning to align the view-specific representations, driving them into the same semantic latent space to minimize the discrepancy in view-specific distributions. In addition, a consensus latent representation is learned by an adaptive fusion block that integrates information from multiple views. The consensus representation is further refined through adversarial learning modeling the transformation of the standard Gaussian distribution to the original data distribution. Moreover, the proposed method incorporates an imputation strategy designed to handle the incomplete multi-view data clustering task. This strategy utilizes both reconstructed samples and cross-view neighbors to impute missing views from the latent space and the original space, thereby preserving clustering information, which ensures the quality and feasibility of the imputed samples. Experimental results on six widely used datasets have verified the competitiveness of the proposed CDARL method against state-of-the-art methods in MVC problems with complete and incomplete multi-view data. Code is available athttps://github.com/xywy220/CDARL-MVC.

Incomplete multi-view clustering (IMVC) has attracted increasing attention in recent years, owing to the prevalence of missing data in real-world multi-view scenarios. Existing imputation-based IMVC methods partially mitigate the impact of missing information but still face three key limitations: (i) overlooking latent structural relationships among samples, which leads to imputed representations deviating from the true distribution; (ii) decoupling imputation from clustering, which reduces the discriminability of the recovered representations; and (iii) exhibiting low efficiency, which makes it difficult to balance recovery quality and inference speed under complex missing scenarios. To address these issues, we propose a Structure-Aware Conditional Diffusion Generation (SACDG) framework. During training, SACDG first models local structural relationships via adaptive neighborhood graphs and injects them as conditional priors into the diffusion model, where a cross-attention mechanism integrates these priors into the noise prediction process to learn structure-aware generative capability. Meanwhile, a semantic distribution alignment module is introduced to leverage pseudo-labels for enforcing cross-view consistency, thereby enhancing semantic discriminability. During inference, SACDG integrates cross-view structural information through cross-view adjacency fusion to guide the reverse denoising trajectory, and employs deterministic DDIM sampling to efficiently and stably recover the representations of missing views. Extensive comparative experiments and ablation studies on multiple benchmark datasets demonstrate that SACDG achieves superior clustering performance and improved efficiency over state-of-the-art methods. Our code is available athttps://github.com/zhangyuanyang21/SACDG.

Spatially resolved transcriptomics (SRT) facilitates the simultaneous acquisition of gene expression profiles, spatial location, and histology images for spatial clustering analysis, providing transformative insights into cellular interactions and the underlying mechanisms of disease progression. Despite the success of existing research in spatial clustering tasks, most methods overlook the information imbalance arising among spots in intra- and inter-modal communication due to insufficient sequencing depth and modality discrepancies. To this end, we propose a novel multi-view information rebalance learning method for SRT data clustering, referred to as MIRL. Specifically, we construct hypergraphs for the gene and histological image modalities and leverage hypergraph neural networks to learn the hypergraph features, which helps mitigate the propagation of intra-modal information imbalance by capturing higher-order interactions among multiple spots, rather than relying solely on pairwise relationships in traditional feature graphs. To enhance the global coordination among spots and the interrelations between features across modalities, we perform intra-modal adaptive fusion of modality-specific hypergraph features and spatial features, followed by cross-modal integration. Furthermore, adaptive reconstruction of the cross-modal heterogeneous graph is employed to rebalance inter-modal information flow associated with pseudo-labels, ensuring more reliable information extraction by alleviating the impact of incorrect heterogeneous negative edges connections through the construction of hypergraph edges. Extensive experimental results demonstrate that the proposed MIRL achieves competitive performance in spatial domain identification compared to other state-of-the-art ones.

Multi-view clustering (MVC) for remote sensing data is a critical and challenging task in Earth observation. Although recent advances in graph neural network (GNN)-based MVC have shown remarkable success, the most prevalent approaches have two major limitations: 1) heavily relying on a predefined yet fixed graph, which limits the performance of clustering because the large number of indistinguishable background samples contained in remote sensing data would introduce noise information and increase structure heterogeneity; 2) ignoring the effect of confusing samples on cluster structure compactness, which leads to fluffy cluster structure and decrease feature discriminability. To address these issues, we propose a Structure-Adaptive Multi-View Graph Clustering method named SAMVGC on remote sensing data which boosts the structure homogeneity and cluster compactness by adaptively learning the graph and cluster structures, respectively. Concretely, we use the geometric structure within the feature embedding space to refine adjacency matrices. The adjacency matrices are dynamically fused with the previous ones to improve the homogeneity and stability of structure information. Additionally, the samples are separated into two categories, including the central (intra-cluster center samples) and the confusing (inter-cluster boundary samples). On the basis, we deploy the contrastive learning paradigm on the central samples within views and the consistent learning paradigm on the confusing samples between views, improving the cluster compactness and consistency. Finally, we conduct extensive experiments on four benchmarks and achieve promising results, well demonstrating the effectiveness and superiority of the proposed method.

Multi-view clustering can partition data samples into their categories by learning a consensus representation in an unsupervised way and has received more and more attention in recent years. However, there is an untrusted fusion problem. The reasons for this problem are as follows: 1) The current methods ignore the presence of noise or redundant information in the view; 2) The similarity of contrastive learning comes from the same sample rather than the same cluster in deep multi-view clustering. It causes multi-view fusion in the wrong direction. This paper proposes a novel multi-view clustering network to address this problem, termed as Trusted Mamba Contrastive Network (TMCN). Specifically, we present a new Trusted Mamba Fusion Network (TMFN), which achieves a trusted fusion of multi-view data through a selective mechanism. Moreover, we align the fused representation and the view-specific representation using the Average-similarity Contrastive Learning (AsCL) module. AsCL increases the similarity of view presentation from the same cluster, not merely from the same sample. Extensive experiments show that the proposed method achieves state-of-the-art results in deep multi-view clustering tasks. The source code is available at https://github.com/HackerHyper/TMCN.

Trustworthy multi-modal learning integrates multiple sources of data reliably. However, the current methods still focus on performance improvement by developing deep multi-modal networks. These approaches frequently encounter challenges due to the inherent non-convex nature of deep neural networks and their vulnerability to local minima, ultimately leading to a diminished ability for generalization. To address this problem, we present a novel curriculum termed the Dynamic SRM Curriculum (DSRMC). Within DSRMC, the deep trustworthy multi-modal networks undergo training with data provided sequentially, progressing from simple to complex samples. This training strategy mimics the human learning process, commencing with fundamental concepts and gradually advancing to tackle more complex and abstract ideas. Building upon DSRMC, we propose an innovative Curriculum Trustworthy Multi-modal Learning (CTML) method. CTML makes it easier to place the learned model in a flatter area, which improves its overall ability for generalization. Comprehensive experiments on three public datasets demonstrate that the proposed CTML performs better than state-of-the-art methods, achieving a maximum improvement of 6.7% on macroF1.

Despite their impressive capabilities, Multimodal Large Language Models (MLLMs) are prone to hallucinations, i.e., the generated content that is nonsensical or unfaithful to input sources. Unlike in LLMs, hallucinations in MLLMs often stem from the sensitivity of text decoder to visual tokens, leading to a phenomenon akin to "amnesia" about visual information. To address this issue, we propose MemVR, a novel decoding paradigm inspired by common cognition: when the memory of an image seen the moment before is forgotten, people will look at it again for factual answers. Following this principle, we treat visual tokens as supplementary evidence, re-injecting them into the MLLM through Feed Forward Network (FFN) as “key-value memory” at the middle trigger layer. This look-twice mechanism occurs when the model exhibits high uncertainty during inference, effectively enhancing factual alignment. Comprehensive experimental evaluations demonstrate that MemVR significantly mitigates hallucination across various MLLMs and excels in general benchmarks without incurring additional time overhead.

Incomplete Multi-View Clustering (IMVC) aims to explore comprehensive representations from multiple views with missing samples. Recent studies have revealed that IMVC methods benefit from Graph Convolutional Network (GCN) in achieving robust feature imputation and effective representation learning. Despite these notable improvements, GCN imputation methods often cause a distribution shift between the imputed and original representations, particularly when the neighbors of the imputed nodes are assigned to different groups. Moreover, GCN learning methods tend to produce homogeneous imputed representations, which blur cluster boundaries and hinder effective discriminative clustering. To remedy these challenges, the Local Refinement and Global Realignment (LRGR) Self-supervised model is proposed for incomplete multi-view clustering, which includes two stages. In the first stage, a local imputed refinement module is designed to enhance the versatility of imputed representations through cross-view contrastive learning guided by view-specific prototypes. In the second stage, a global realignment module is introduced to achieve semantic consistency across views, alleviating distribution shifts by leveraging pseudo-labels and their corresponding confidence scores as guidance. Experiments on five widely used multi-view datasets demonstrate the competitiveness and superiority of our method compared to state-of-the-art approaches.

Spatial transcriptomics, comprising spatial location and high-throughput gene expression information, provides revolutionary insights into disease discovery and cellular evolution. Spatial transcriptomic clustering, which pinpoints distinct spatial domains within tissues, reveals cellular interactions and enhances our understanding of the intricate architecture of tissues. Existing methods typically construct spatial graphs using a static radius based on spatial coordinates, which hinders the accurate identification of spatial domains and complicates the precise partitioning of boundary nodes within clusters. To address this issue, we introduce a novel spatially resolved transcriptomics data clustering network (TSstc). Specifically, we employ a tailored spatial-scale modulation approach, constructing different spatial graphs incrementally as the radius of the spatial domain expands, and a Spatiality-Aware Sampling (SAS) strategy is proposed to aggregate node representations by considering the spatial dependencies between spots. We then use GCN encoders to learn gene embedding with gene graph and multiple spatial embeddings with spatial graphs. During training, we incorporate cross-view correlation-based tailored spatial regularization constraints to preserve high-quality neighbor relationships across spatial embeddings at different scales. Finally, a zero-inflated negative binomial model is utilized to capture the global probability distribution of gene expression profiles. Extensive experimental results demonstrate that our approach surpasses existing state-of-the-art methods in clustering tasks and related downstream applications.

Partially view-aligned clustering (PVC) has emerged as a critical area in multi-view clustering, addressing the inherent instance misalignment across views during data collection. The primary challenge of PVC is accurately establishing correspondences between cross-view samples. The Banzhaf index in cooperative game theory serves as an effective tool for modeling complex relationships between multi-view samples by quantifying the marginal contributions of coalition members to collaborative benefits. To this end, we propose a Banzhaf Index-driven cross-view aligNment method, dubbed BIN, which systematically evaluates each view sample's contribution to joint decision-making within a game-theoretic framework. This approach overcomes the limitations of existing PVC methods reliant on prior alignment information and enhances the robustness of multi-view matching. Specifically, we model multi-view samples as players in a cooperative game and quantify their interactions using a payoff model. Simultaneously, we propose a dual-loss constraint: (1) Banzhaf gain loss, which dynamically captures the marginal contribution of key cross-view sample pairs to reinforce associations; (2) contrast loss, which applies exclusion constraints in the feature space to suppress interference from weakly correlated samples. Together, these losses form an effective optimization mechanism. This game-theoretic approach adaptively learns sample correspondences without pre-alignment and ensures robust matching in complex misalignment scenarios. Extensive experiments demonstrate that our method achieves competitive performance against eight state-of-the-art PVC algorithms.

Recent diffusion-based methods have shown strong ability in the depth estimation task, but they largely overlook the rich textual priors embedded in pretrained diffusion models that can enhance both performance and robustness in diverse scenes. In this paper, we propose TPDepth, a diffusion-based, affine-invariant monocular depth estimator that incorporates textual semantics via a Text-Prompted ControlNet. While directly injecting text into the diffusion U-Net can cause the network to over-attend to local semantic cues and compromise global structural modeling, TPDepth processes textual features through a separate ControlNet branch, allowing semantic information to be incorporated without disrupting the spatial reasoning pipeline. Prompt-conditioned features are modulated by an Adaptive Control Scale Module(ACSM) and injected into decoder of the diffusion UNet with skip connections. The model is fine-tuned with a fixed timestep for deterministic prediction. TPDepth achieves state-of-the-art results on NYUv2, KITTI, and ScanNet, and demonstrates competitive performance on two additional zero-shot benchmarks using only 61K training images. Code and models can be found on our https://github.com/Lioely/TPDepth project page.

Existing multi-view clustering methods employ various strategies to address data-level sparsity and view-level dynamic fusion. However, we identify a critical yet overlooked issue: varying sparsity across views. Cross-view sparsity variations lead to encoding discrepancies, heightening sample-level semantic heterogeneity and making view-level dynamic weighting inappropriate. To tackle these challenges, we propose Adaptive Sparse Autoencoders for Multi-View Clustering (SparseMVC), a framework with three key modules. Initially, the sparse autoencoder probes the sparsity of each view and adaptively adjusts encoding formats via an entropy-matching loss term, mitigating cross-view inconsistencies. Subsequently, the correlation-informed sample reweighting module employs attention mechanisms to assign weights by capturing correlations between early-fused global and view-specific features, reducing encoding discrepancies and balancing contributions. Furthermore, the cross-view distribution alignment module aligns feature distributions during the late fusion stage, accommodating datasets with an arbitrary number of views. Extensive experiments demonstrate that SparseMVC achieves state-of-the-art clustering performance. Our framework advances the field by extending sparsity handling from the data-level to view-level and mitigating the adverse effects of encoding discrepancies through sample-level dynamic weighting. The source code is publicly available at https://github.com/cleste-pome/SparseMVC.

Most existing multi-view clustering methods aim to generate a consensus partition across all views, based on the assumption that all views share the same sample arrangement. However, in real-world scenarios, the collected data across different views is often unsynchronized, making it difficult to ensure consistent sample correspondence between views. To address this issue, we propose a scalable sample-alignment-based multi-view clustering method, referred to as SSA-MVC. Specifically, we first employ a cluster-label matching (CLM) algorithm to select the view whose clustering labels best match those of the others as the benchmark view. Then, for each of the remaining views, we construct representations of non-aligned samples by computing their similarities with aligned samples. Based on these representations, we build a similarity graph between the non-aligned samples of each view and those in the benchmark view, which serves as the alignment criterion. This alignment criterion is then integrated into a late-fusion framework to enable clustering without requiring aligned samples. Notably, the learned sample alignment matrix can be used to enhance existing multi-view clustering methods in scenarios where sample correspondence is unavailable. The effectiveness of the proposed SSA-MVC algorithm is validated through extensive experiments conducted on eight real-world multi-view datasets.

Simple multiple kernel k-means (SMKKM) introduces a new minimization-maximization learning paradigm for multi-view clustering and makes remarkable achievements in some applications. As one of its variants, localized SMKKM (LSMKKM) is recently proposed to capture the variation among samples, focusing on reliable pairwise samples, which should keep together and cut off unreliable, farther pairwise ones. Though demonstrating effectiveness, we observe that LSMKKM indiscriminately utilizes the variation of each sample, resulting in unsatisfying clustering performance. To overcome this limitation, we propose a sample adaptive localized SMKKM (SAL-SMKKM) algorithm where the weight of the local alignment for each sample can be adaptively adjusted, resulting in a more challenging tri-level minimization-minimization-maximization. To deal with it, we reformulate it into a minimization problem of an optimal function characterized by minimization-maximization dynamics, prove its differentiability, and develop a reduced gradient descent method to optimize it. We then theoretically analyze the clustering performance of the proposed SAL-SMKKM by deriving its generalization error bound. In addition, we empirically evaluate the clustering performance of the proposed SAL-SMKKM on several benchmark datasets. Experiment results clearly indicate that proposed algorithms consistently outperform state-of-the-art ones. Finally, we apply the proposed SAL-SMKKM to the multi-modal parcellation of the human cerebral cortex, which is essential and helpful to understanding brain organization and function. As seen, SAL-SMKKM achieves accurate parcellation in an automatic and objective manner without any manual intervention, which once again demonstrates its validity and effectiveness in practical applications.

Antimicrobial peptides (AMPs) are crucial in addressing the global crisis of bacterial resistance. However, there are still significant limitations in existing methods on de novo AMPs design, especially in designing AMPs with desirable physicochemical properties for specific bacterial pathogens. In this study, we propose a novel generative framework for designing pathogen-targeted antimicrobial peptides with programmable physicochemical properties. More specifically, a conditional Variational Autoencoder is first pretrained for generating AMPs with editable physicochemical properties. We then develop a conditional diffusion model to learn hidden representations of AMPs for targeting pathogens of interest, and construct corresponding MIC predictors for specific bacterial strains. Through comprehensive simulation experiments, we demonstrate that the proposed framework outperforms most existing models in terms of antimicrobial efficacy against specific bacterial targets. Moreover, through systematic screening and analysis, we have identified two star AMPs for each of the two target bacterial species (i.e., E. coli or S. aureus), both of which exhibit excellent performance in antibacterial activity, hemolytic properties, toxicity profiles, etc. Overall, this study provides the key technological support for developing next-generation intelligent platforms for antimicrobial agents design.

Video-based facial expression recognition (VFER) is challenging due to variations caused by cultural background and expression camouflage. To tackle these problems, researchers introduced eye movement signals to complement visual information. However, existing methods either require expensive devices to capture high-quality eye movements or can only extract low-quality eye movements visually, making them ineffective in the real world. To address this, we propose an eye movement-instructed VFER (EM-VFER) that leverages high-quality eye movements to instruct the visual learning, obtaining robust performance without requiring costly devices during inference. Specifically, our EM-VFER operates in two stages: the high-quality eye movement pre-training stage and the eye movement-instructed video fine-tuning stage. In the pre-training, we compile an Eye-behavior-aided Multimodal Emotion Recognition (EMER) dataset and use it to train a multimodal Transformer. During the fine-tuning, we propose a novel progressive eye movement-instructed learning to take better advantage of the prior knowledge about high-quality eye movement signals from EMER. The instructed fine-tuning model could then make more robust predictions on downstream facial expression datasets. We evaluate our approach on three macroexpression datasets (DFEW, MAFW and Aff-wild2) and two micro-expression datasets (CASME III and CASME II). The results demonstrate that EM-VFER significantly outperforms existing methods. The code will be available.

The rapid advancement of single-cell sequencing technology has generated vast amounts of multi-omics data, presenting unprecedented opportunities for single-cell multi-omics clustering analysis. However, existing single-cell clustering algorithms focus on extracting shared representations, overlooking the interactions and correlations among cells. This oversight inevitably leads to biased or confounded cell clustering results. In this paper, we propose a cell similarity purified adaptive fusion network for single-cell multi-omics clustering, named scSPAF, which adopts a multi-level fusion approach to thoroughly explore the consistency and complementarity of omics data. Specifically, we design a cell similarity purification module to accurately incorporate neighborhood information among cells into cell features, thereby purifying the latent representation that reflects cell correlations. In addition, we align attribute features from different omics to extract the consistent representation across omics. Simultaneously, by employing an adaptive fusion mechanism, we integrate representations of omics-specific and the consistent representation across omics to generate more discriminative representations of omics, further enhancing the clustering performance. Experimental results obtained from six real-world datasets demonstrate the superiority of the scSPAF algorithm when compared with other state-of-the-art methods.

Hyperspectral band selection, aimed at identifying key spectral bands from the original image, is crucial for reducing dimensionality and enhancing computational efficiency in hyperspectral image (HSI) analysis. Graph learning-based methods have attracted considerable attention due to their efficiency in representing structural correlations between bands and their powerful capability to extract features. However, existing methods have limitations in utilizing spatial relationships among bands and learning their discriminative characteristics. To address these limitations, we propose a Diversity Learning Guided Dual Graph Autoencoder (DLG-DGAE) for unsupervised hyperspectral band selection. In our framework, we integrate a Dual Graph Autoencoder (DGAE) module designed to extract information from both the spatial and spectral relationships among bands, thus fully capturing the structural similarity of the bands. Additionally, we introduce a Spectral Diversity Learning (SDL) strategy to reduce redundant information in the latent representation and enhance the discriminative properties of each band. In the final step, we proceed to cluster the fused latent embeddings. Within each cluster, we select the band exhibiting the highest information entropy as the representative band. Through extensive experimentation on three publicly available datasets, our results consistently indicate that the proposed method surpasses other state-of-the-art techniques. The code is available athttps://github.com/fengwe1/DLG-DGAE.

Multi-view clustering (MvC) utilizes information from multiple views to uncover the underlying structures of data. Despite significant advancements in MvC, mitigating the impact of missing samples in specific views on the integration of knowledge from different views remains a critical challenge. This paper proposes a novel Mask-informed Deep Contrastive Incomplete Multi-view Clustering (Mask-IMvC) method, which elegantly identifies a view-common representation for clustering. Specifically, we introduce a mask-informed fusion network that aggregates incomplete multi-view information while considering the observation status of samples across various views as a mask, thereby reducing the adverse effects of missing values. Additionally, we design a prior knowledge-assisted contrastive learning loss that boosts the representation capability of the aggregated view-common representation by injecting neighborhood information of samples from different views. Finally, extensive experiments are conducted to demonstrate the superiority of the proposed Mask-IMvC method over state-of-the-art approaches across multiple MvC datasets, both in complete and incomplete scenarios. The demo code for our work will be publicly available at https://github.com/guanyuezhen/Mask-IMvC.

In recent years, Incomplete Multi-View Clustering (IMVC) has become an important and challenging task. Although several methods have been proposed to address IMVC, they still have the following drawbacks: i) Due to the presence of missing samples in the views, clustering prototypes obtained from different views may have positional deviations, leading to inaccurate positioning of cluster centers, thus affecting the accuracy of clustering results. ii) Repair strategies based on cross-view prediction and adversarial generation have high computational costs and heavily rely on model performance. Neighbor-based repair strategies may result in inaccurate neighbor selection due to the presence of noise. iii) Models learned solely from complete data often perform better than models learned from both complete and incomplete data, especially when there are semantic differences between the repaired data and the missing data. To address the aforementioned issues, this paper proposes a Dynamic Imputation and Triple Alignment with Dual-Optimization for Deep Incomplete Multi-View Clustering (DITA-IMVC). Specifically, by accurately representing advanced features, we propose a cross-view dynamic structure learning strategy for missing view repair, where the dynamic structural relationships between high-semantic features within each view are calculated to obtain highly related samples from different views. To address positional deviations from different views, we propose a triple cross-view alignment with prototype, feature, and clustering assignment, which preserves the consistency among different views. Finally, we design a dual-optimization process for both complete view features and repaired features via alternating iterations to fully utilize the incomplete view data, thereby improving clustering performance. To demonstrate the effectiveness of our DITA-IMVC, extensive experiments conducted on different standard datasets show that it yields superior clustering results compared to existing methods.

Despite advances in deep learning for remote sensing building extraction (RSBE), Multi-target Domain RSBE (MD-RSBE) remains challenging, as it requires transferring knowledge from a labeled source domain to multiple unlabeled target domains, with domain shifts in texture, style, and semantics. Existing domain adaptation (DA) and generalization (DG) methods face significant limitations: DA requires target-domain training, while DG needs multi-source training, leading to high training costs and low generalization in practical MD-RSBE scenarios. To address this, we propose a Multi-view Augmented Single-source Domain Generalization (MASDG) method, which effectively mitigates domain shifts across RS source and target domains for robust MD-RSBE performance by enriching the diversity of the source domain through multi-view augmentation and enforcing semantic consistency. Specifically, MASDG consists of three key components: Texture-level Domain Augmentation (TDA) module, Style-level Domain Augmentation (SDA) module and Semantic-invariant Representation Learning (SRL). To mitigate texture-level domain shift, TDA first introduces parameter-optimized multi-layer random convolution to modify the texture of source image, generating texture-augmented image pairs for simulating real-world texture diversity across various RS domains. Then, with each image pair from TDA, SDA employs two paralleled encoders, namely the general feature encoder and the batch-guided style encoder, to formulate multi-view building features, further mitigating style-level domain shift. Finally, SRL ensures semantic-invariant representation learning via a dual mechanism, including multi-view segmentation loss and semantic consistency loss. The former generates predictions from diverse feature views (original, texture-augmented, style-augmented, etc.), while the latter performs semantic alignment by minimizing distribution discrepancies among predictions, bridging semantic inconsistency to enable robust segmentation. Extensive experiments across three different MD-RSBE settings with 7 different target domains demonstrate that our MASDG outperforms existing state-of-the-art methods by a significant margin.

Multimodal remote sensing image segmentation (MRSIS) is important for intelligent remote sensing image (RS) interpretation, which encompasses three distinct tasks: semantic segmentation, instance segmentation, and panoptic segmentation. Existing methods typically address individual tasks with specialized models, limiting generalization and real-world applicability. Multi-task learning approaches have introduced separated task heads to unify tasks, yet we identify two key challenges when directly applying them to MRSIS: (1) the modality gap, arising from semantic discrepancies and granularity discrepancies across RS modalities, and (2) the task gap, due to varying preferences in learning different segmentation tasks. To overcome these challenges, we propose PMTSeg—a novel Prompt-driven Multimodal Transformer for task-adapted MRSIS. PMTSeg integrates three key components: (1) Task-common Multimodal Affinity Approximation (TMAA), (2) Task-common Multi-scale Semantic Fusion (TMSF), and (3) a unified Prompt-driven Segmentation Head (PSH). First, TMAA addresses the modality gap by approximating inter-modal affinity matrices, extracting task-common features across modalities and aligning semantic information. Then, TMSF further integrates these features using the scale-matched fusion at multiple scales to produce enriched, multi-scale task-common features. Moreover, to address the task gap, the PSH leverages task-adapted text prompts and task-adapted contrastive loss to model relationships across tasks, enabling adaptive optimization for robust and universal MRSIS performance. Extensive experiments on three MRSIS datasets—VALID, SEMCITY TOULOUSE, and UBCV2—demonstrate that PMTSeg significantly surpasses state-of-the-art methods in all three segmentation tasks, offering a unified and accurate solution to MRSIS.

The research on identifying drug-disease associations (DDAs) is widely used in scenarios such as drug development, clinical decision-making, and drug repurposing, holding significant biological and medical significance. Existing methods for drug-disease association prediction have achieved decent performance, they primarily rely on simplistic drug-disease association graphs or similarity graphs. These methods often struggle to capture the high-order correlations of complex multimodal data, limiting their ability to handle the complexity of data associations effectively. In addition, real drug-disease associations are highly sparse, posing a significant challenge to prediction accuracy. To tackle these issues, we propose a general hypergraph neural network framework for drug-disease association prediction based on hierarchical contrastive learning and cross-attention learning. It leverages hypergraph neural networks to learn representations of drugs and diseases carrying high-order correlations and strengthens representation quality using interactive attention learning and hierarchical contrastive learning. Meanwhile, the $\lambda$-weighted loss function is utilized to adapt to the high sparsity property of real drug-disease associations during model training and improve prediction performance. Extensive experiments demonstrate that DD-HGNN$^+$ surpasses other state-of-the-art methods in predicting drug-disease associations and further validation through case studies on Leukemia and Colorectal Neoplasms underscores its reliability.

Multi-view clustering (MVC) for remote sensing data has demonstrated significant potential in Earth observation, given its ability to aggregate multi-source information without relying on labels. Despite achieving compelling results through the combination of deep encoders and contrastive learning, existing algorithms still face two limitations: inadequate exploration of diverse spatial relationships and inability to guide the selection of sample pairs leads to blind sampling, both of which lead to suboptimal clustering performance. To tackle these challenges, we propose a sampling enhanced contrastive multi-view clustering method for remote sensing data, namely SEC-LSRM. The proposed method incorporates long- and short-range information mining to enhance clustering performance. By aggregating shortrange information extracted through autoencoders and longrange information obtained via graph autoencoders, our method improves the sampling quality of positive and negative sample pairs. To render the extracted features more compact, a multiview correlation reduction strategy is devised to filter out irrelevant information. With the extracted comprehensive features, an adaptive sampling strategy is designed to obtain high-quality positive and negative samples. Subsequently, we select positive and negative sample pairs based on these affinity matrices with idempotence and block diagonal constraints. Moreover, we integrate the optimization of these sample pairs and contrastive learning within the same framework to achieve iterative updates of both. Experiments conducted on multiple multi-view remote sensing datasets illustrate that our proposed SEC-LSRM method achieves excellent and reliable clustering performance.

Graph-based multi-view clustering methods have demonstrated satisfying performance by effectively capturing relationships among data samples. However, most existing methods primarily emphasize direct pairwise relationships, neglecting the exploration of high-order correlations present within each view. To this end, a novel approach, called multiview clustering via high-order bipartite graph learning and tensor low-rank representation (HBGTLRR), is proposed. Specifically, we first construct high-order bipartite graphs to capture latent relationships and concatenate them into a tensor. By applying tensor nuclear norm (TNN) minimization, we obtain a low-rank representation that reduces noise and preserves high-order consistency. Subsequently, a consensus graph is constructed by adaptively fusing the high-order bipartite graphs with corresponding weights, and then a Laplacian low-rank constraint is imposed on it to effectively capture the intrinsic data structure. Finally, extensive experimental results show that HBGTLRR significantly outperforms existing methods, thereby validating the effectiveness of our proposed method.

Multi-view clustering (MVC) exploits the information captured from diverse views to partition data into different groups and attracts much attention recently. Despite significant progress, most MVC methods fuse multi-view information via one-stage fusion while neglecting the merits of multi-stage fusion which causes insufficient in utilizing rich information within data and therefore degrades the clustering performance. To this end, designing a functional framework that can fully exploit multi-view information becomes a key challenge in multi-view clustering research. In this paper, we propose a novel multi-stage fusion method, which elegantly unifies the late and early fusion into one unified framework, to capture sufficient information underlying the multi-view data and to effectively reduce the effect of low-quality views. Specifically, we construct a low dimensional latent representation from multi-view data by learning proper correlation among multi-view data in the early fusion stage. The late fusion establishes a new optimal combinational data partition from base partitions constructed by spectral clustering, which suppresses the influence of low-quality basic partitions. Then we couple the low dimensional latent representation with the learned combinational data partition to share the same cluster structure by$k$-means and maximization alignment. As a result, we collaboratively learn an accurate and robust partition representation for the following clustering task. Besides, the late fusion and early fusion are jointly learned to achieve mutual collaboration for better performance. Finally, an alternating optimization algorithm is designed to solve the resultant optimization problem. Extensive experiments conducted on eight datasets show the superiority of our method in terms of effectiveness and efficiency.

Hyperspectral images with high spectral resolution provide new insights into recognizing subtle differences in similar substances. However, object detection in hyperspectral images faces significant challenges in intra- and inter-class similarity due to the spatial differences in hyperspectral inter-bands and unavoidable interferences, e.g., sensor noises and illumination. To alleviate the hyperspectral inter-bands inconsistencies and redundancy, we propose a novel network termedSpectralDiscrepancy andCross-Modal semantic consistency learning (SDCM), which facilitates the extraction of consistent information across a wide range of hyperspectral bands while utilizing the spectral dimension to pinpoint regions of interest. Specifically, we leverage a semantic consistency learning (SCL) module that utilizes inter-band contextual cues to diminish the heterogeneity of information among bands, yielding highly coherent spectral dimension representations. On the other hand, we incorporate a spectral gated generator (SGG) into the framework that filters out the redundant data inherent in hyperspectral information based on the importance of the bands. Then, we design the spectral discrepancy aware (SDA) module to enrich the semantic representation of high-level information by extracting pixel-level spectral features. Extensive experiments on two hyperspectral datasets demonstrate that our proposed method achieves state-of-the-art performance when compared with other ones.

Human-centric Emotional Video Captioning (H-EVC) aims to generate fine-grained, emotion-related sentences for human-based videos, enhancing the understanding of human emotions and facilitating human-computer emotional interaction. However, existing video captioning methods often overlook subtle emotional clues and interactions in videos. As a result, the generated captions frequently lack emotional information. To address this, we proposeEmotion-orientedCross-modalPrompting andAlignment (ECPA), which improves HEVC accuracy by modeling fine-grained visual-textual emotion clues. Using large foundation models, ECPA introduces two learnable prompting strategies: visual emotion prompting (VEP) and textual emotion prompting (TEP), along with an emotion-oriented cross-modal alignment (ECA) module. VEP uses two levels of visual prompts,i.e., emotion recognition (ER) and action unit (AU), to focus on both coarse and fine visual emotional features. TEP devise two-level learnable textual prompts,i.e., sentence-level emotional tokens and word-level masked tokens to capture global and local textual emotion representations. ECA introduces another two levels of emotion-oriented prompt alignment learning mechanisms: the ER-sentence level and the AU-word level alignment losses. Both enhance the model's ability to capture and integrate both global and local cross-modal emotion semantics, thereby enabling the generation of fine-grained emotional linguistic descriptions in video captioning. Experiments show ECPA significantly outperforms state-of-the-art methods on various H-EVC datasets (relative improvements of 9.98%, 5.72%, 4.46%, 24.52% on MAFW, and 12.82%, 20.27%, 4.22%, 5.01% on EmVidCap across four evaluation metrics) and supports zero-shot tasks on MSVD and MSRVTT, demonstrating strong applicability and generalization.

Multi-view clustering, which identifies shared semantics from different perspectives and classifies data samples into distinct categories using unsupervised methods, is gaining increasing interest. This task primarily focuses on learning consistent multi-view feature representations and clustering labels. Current approaches for achieving consistent multi-view feature representations often use techniques such as cascading, weight fusion, and attention mechanism fusion. These methods reconstruct features based on original low-level features via encoder-decoder, which often contain visual private information, leading to misleading feature representations. Furthermore, in the clustering label learning process, many methods use a two-stage approach: first, they achieve consistent feature representations, and then they apply hard labeling methods like K-means or spectral clustering to obtain clustering labels. Single-stage methods typically derive consistent labels through a linear coding layer based on consistent representation learning. These methods do not fully utilize the multi-view view semantic information, and consistent representation learning may be impaired when some low-quality views are present, leading to the generation of inaccurate semantic labels. To address these issues, we propose a Self-supervised Semantic Soft Label Learning Network for Deep Multi-view Clustering. Specifically, we introduce a consensus high-level feature learning module that uses a shared MLP layer to transform low-level features into a high-level feature space. To enhance the consistency between high-level features from different views, we maximize mutual information between these features and introduce the U-Projection module, which improves the expressive power of the consensus feature via resampling the features and concatenating the fused features before and after sampling operations. Additionally, we propose a self-supervised semantic label learning module that employs a dual-branch approach to independently learn consistent view-specific semantic labels through contrastive learning, while deriving view-consensus semantic labels from shared high-level features extracted from multiple views. Finally, KL divergence is used to align the view-consensus labels with the view-specific labels. A series of extensive experiments have shown that our approach yields superior clustering results compared to existing techniques.

Multi-view hashing is a crucial technology for multimedia retrieval because it transforms heterogeneous data from many viewpoints into binary hash codes. However, the existing approaches focus mostly on the complementarity among multiple views while being without confidence fusion. Furthermore, redundant noise is present in the single-view data in real-world application contexts. We present an innovativeAdaptive Confidence Multi-View Learning(ACMVL) method to perform confidence fusion and remove extraneous noise. Initially, a confidence network is constructed to eliminate noise data and extract useful information from various single-view features. Moreover, an adaptive confidence multi-view network is utilized to quantify the confidence of each view and further fuse multiple view features using a weighted summation. Here, we propose anAutomatic View Confidence Metric(AVCM) as a score for evaluating the confidence of views. Finally, to improve the semantic representation of the fused feature, a dilation network is created. Based on ACMVL, we introduce a novelAdaptive Confidence Multi-View Hashing(ACMVH) method. To our knowledge, we are the pioneers in using confidence learning for multimedia retrieval. Comprehensive experiments on three publicly available datasets demonstrate that our ACMVH outperforms the state-of-the-art methods (maximum improvement of 3.24% on mAP).

Graph regularized nonnegative matrix factorization (GNMF) has been widely used in data representation due to its excellent dimensionality reduction. When it comes to clustering polluted data, GNMF inevitably learns inaccurate representations, leading to models that are unusually sensitive to outliers in the data. For example, in a face dataset, obscured by items such as a mask or glasses, there is a high probability that the graph regularization term incorrectly describes the association relationship for that sample, resulting in an incorrect elicitation in the matrix factorization process. In this article, a novel self-initiated unsupervised subspace learning method named robust nonnegative matrix factorization with self-initiated multigraph contrastive fusion (RNMF-SMGF) is proposed. RNMF-SMGF is capable of creating samples with different angles and learning different graph structures based on these different angles in a self-initiated method without changing the original data. In the process of subspace learning guided by graph regularization, these different graph structures are fused into a more accurate graph structure, along with entropy regularization, $L_{2,1/2}$ -norm constraints to facilitate the robust learning of the proposed model and the formation of different clusters in the low-dimensional space. To demonstrate the effectiveness of the proposed model in robust clustering, we have conducted extensive experiments on several benchmark datasets and demonstrated the effectiveness of the proposed method. The source code is available at: https://github.com/LstinWh/RNMF-SMGF/.

Multiview data, characterized by rich features, are crucial in many machine learning applications. However, effectively extracting intraview features and integrating interview information present significant challenges in multiview learning (MVL). Traditional deep network-based approaches often involve learning multiple layers to derive latent. In these methods, the features of different classes are typically implicitly embedded rather than systematically organized. This lack of structure makes it challenging to explicitly map classes to independent principal subspaces in the feature space, potentially causing class overlap and confusion. Consequently, the capability of these representations to accurately capture the intrinsic structure of the data remains uncertain. In this article, we introduce an innovative multiview representation learning (MVRL) by maximizing two information-theoretic metrics: intraview coding rate reduction and interview mutual information. Specifically, in the intraview representation learning, we aim to optimize feature representations by maximizing the coding rate difference between the entire dataset and individual classes. This process expands the feature representation space while compressing the representations within each class, resulting in more compact feature representations within each viewpoint. Subsequently, we align and fuse these view-specific features through space transformation and cross-sample fusion to achieve consistent representation across multiple views. Finally, we maximize information transmission to maintain consistency and correlation among data representations across views. By maximizing mutual information between the consensus representations and view-specific representations, our method ensures that the learned representations capture more concise intrinsic features and correlations among different views, thereby enhancing the performance and generalization ability of MVL. Experiments show that the proposed methods have achieved excellent performance.

Multiview clustering (MVC) has gained significant attention as it enables the partitioning of samples into their respective categories through unsupervised learning. However, there are a few issues as follows: 1) many existing deep clustering methods use the same latent features to achieve the conflict objectives, namely, reconstruction and view consistency. The reconstruction objective aims to preserve view-specific features for each individual view, while the view-consistency objective strives to obtain common features across all views; 2) some deep embedded clustering (DEC) approaches adopt view-wise fusion to obtain consensus feature representation. However, these approaches overlook the correlation between samples, making it challenging to derive discriminative consensus representations; and 3) many methods use contrastive learning (CL) to align the view's representations; however, they do not take into account cluster information during the construction of sample pairs, which can lead to the presence of false negative pairs. To address these issues, we propose a novel multiview representation learning network, called anchor-sharing and clusterwise CL (CwCL) network for multiview representation learning. Specifically, we separate view-specific learning and view-common learning into different network branches, which addresses the conflict between reconstruction and consistency. Second, we design an anchor-sharing feature aggregation (ASFA) module, which learns the sharing anchors from different batch data samples, establishes the bipartite relationship between anchors and samples, and further leverages it to improve the samples' representations. This module enhances the discriminative power of the common representation from different samples. Third, we design CwCL module, which incorporates the learned transition probability into CL, allowing us to focus on minimizing the similarity between representations from negative pairs with a low transition probability. It alleviates the conflict in previous sample-level contrastive alignment. Experimental results demonstrate that our method outperforms the state-of-the-art performance.

Clustering has attracted more and more attention as one of the most fundamental techniques in the field of unsupervised learning. To deal with nonlinear problems, clustering methods have been extended to the kernel version. As a traditional kernel clustering algorithm, multiple kernel k-means (MKKM) aims to learn clustering results from a consensus kernel obtained by combining a set of predefined kernels optimally. However, we observe that the existing MKKM algorithm and its variants insufficiently consider the noise that existed in kernel space and the underlying structure of kernelized data points. To this end, we propose a novel smooth MKKM via underlying graph filtering (SMKKM-UGF) to learn the smooth representations of kernelized data points through their nearby nodes in the underlying graph. In particular, different from the common graph filter, we jointly update the graph filter while learning the smooth kernel, so that the graph filter can be guaranteed to adapt to the updating kernel space constantly. Besides, an iterative algorithm with proven convergence is designed to solve the resultant optimization problem. Extensive experiments have been performed on numerous benchmark datasets, whose results prove the superiority of the proposed SMKKM-UGF compared to the other state-of-the-art clustering methods. The demo code of this work is publicly available at https://github.com/wqyang23/SMKKM-UGF.git.

Anchor technology is popularly employed in multi-view subspace clustering (MVSC) to reduce the complexity cost. However, due to the sampling operation being performed on each individual view independently and not considering the distribution of samples in all views, the produced anchors are usually slightly distinguishable, failing to characterize the whole data. Moreover, it is necessary to fuse multiple separated graphs into one, which leads to the final clustering performance heavily subject to the fusion algorithm adopted. What is worse, existing MVSC methods generate dense bipartite graphs, where each sample is associated with all anchor candidates. We argue that this dense-connected mechanism will fail to capture the essential local structures and degrade the discrimination of samples belonging to the respective near anchor clusters. To alleviate these issues, we devise a clustering framework named SL-CAUBG. Specifically, we do not utilize sampling strategy but optimize to generate the consensus anchors within all views so as to explore the information between different views. Based on the consensus anchors, we skip the fusion stage and directly construct the unified bipartite graph across views. Most importantly, norm and Laplacian-rank constraints employed on the unified bipartite graph make it capture both local and global structures simultaneously. norm helps eliminate the scatters between anchors and samples by constructing sparse links and guarantees our graph to be with clear anchor-sample affinity relationship. Laplacian-rank helps extract the global characteristics by measuring the connectivity of unified bipartite graph. To deal with the nondifferentiable objective function caused by norm, we adopt an iterative re-weighted method and the Newton's method. To handle the nonconvex Laplacian-rank, we equivalently transform it as a convex trace constraint. We also devise a four-step alternate method with linear complexity to solve the resultant problem. Substantial experiments show the superiority of our SL-CAUBG.

Incomplete multi-view clustering has attracted much attention due to its ability to handle partial multi-view data. Recently, similarity-based methods have been developed to explore the complete relationship among incomplete multi-view data. Although widely applied to partial scenarios, most of the existing approaches are still faced with two limitations. Firstly, fusing similarities constructed individually on each view fails to yield a complete unified similarity. Moreover, incomplete similarity generation may lead to anomalous similarity values with column sum constraints, affecting the final clustering results. To solve the above challenging issues, we propose a Sample-level Cross-view Similarity Learning (SCSL) method for Incomplete Multi-view Clustering. Specifically, we project all samples to the same dimension and simultaneously construct a complete similarity matrix across views based on the inter-view sample relationship and the intra-view sample relationship. In addition, a simultaneously learning consensus representation ensures the validity of the projection, which further enhances the quality of the similarity matrix through the graph Laplacian regularization. Experimental results on six benchmark datasets demonstrate the ability of SCSL in processing incomplete multi-view clustering tasks. Our code is publicly available at https://github.com/Tracesource/SCSL.

Hyperspectral image (HSI) clustering is gaining considerable attention owing to recent methods that overcome the inefficiency and misleading results from the absence of supervised information. Contrastive learning methods excel at existing pixel-level and superpixel-level HSI clustering tasks. The pixel-level contrastive learning method can effectively improve the ability of the model to capture fine features of HSI but requires a large time overhead. The superpixel-level contrastive learning method utilizes the homogeneity of HSI and reduces computing resources; however, it yields rough classification results. To exploit the strengths of both methods, we present a pixel–superpixel contrastive learning and pseudo-label correction (PSCPC) method for the HSI clustering. PSCPC can reasonably capture domain-specific and fine-grained features through superpixels and the comparative learning of a small number of pixels within the superpixels. To improve the clustering performance of superpixels, this paper proposes a pseudo-label correction module that aligns the clustering pseudo-labels of pixels and superpixels. In addition, pixel-level clustering results are used to supervise superpixel-level clustering, improving the generalization ability of the model. Extensive experiments demonstrate the effectiveness and efficiency of PSCPC.

Hyperspectral image (HSI) clustering is a challenging task due to its high complexity. Despite subspace clustering shows impressive performance for HSI, traditional methods tend to ignore the global-local interaction in HSI data. In this study, we proposed a multi-level graph subspace contrastive learning (MLGSC) for HSI clustering. The model is divided into the following main parts. Graph convolution subspace construction: utilizing HSI’s spectral and texture feautures to construct two graph convolution views. Local-global graph representation: local graph representations were obtained by step-by-step convolutions and a more representative global graph representation was obtained using an attention-based pooling strategy. Multi-level graph subspace contrastive learning: multi-level contrastive learning was conducted to obtain local-global joint graph representations, to improve the consistency of the positive samples between views, and to obtain more robust graph embeddings. Specifically, graph-level contrastive learning is used to better learn global representations of HSI data. Node-level intra-view and inter-view contrastive learning is designed to learn joint representations of local regions of HSI. The proposed model is evaluated on four popular HSI datasets: Indian Pines, Pavia University, Houston, and Xu Zhou. The overall accuracies are 97.75%, 99.96%, 92.28%, and 95.73%, which significantly outperforms the current state-of-the-art clustering methods.

Spatial transcriptomics provides revolutionary insights into cellular interactions and disease development mechanisms by combining high-throughput gene sequencing and spatially resolved imaging technologies to analyze genes naturally associated with spatially variable tissue genes. However, existing methods typically map aggregated multi-view features into a unified representation, ignoring the heterogeneity and view independence of genes and spatial information. To this end, we construct a heterogeneous Graph guided Contrastive Learning (stGCL) for aggregating spatial transcriptomics data. The method is guided by the inherent heterogeneity of cellular molecules by dynamically coordinating triple-level node attributes through comparative learning loss distributed across view domains, thus maintaining view independence during the aggregation process. In addition, we introduce a cross-view hierarchical feature alignment module employing a parallel approach to decouple spatial and genetic views on molecular structures while aggregating multi-view features according to information theory, thereby enhancing the integrity of inter- and intra-views. Rigorous experiments demonstrate that stGCL outperforms existing methods in various tasks and related downstream applications.

Multi-view clustering (MVC) constitutes a distinct approach to data mining within the field of machine learning. Due to limitations in the data collection process, missing attributes are frequently encountered. However, existing MVC methods primarily focus on missing instances, showing limited attention to missing attributes. A small number of studies employ the reconstruction of missing instances to address missing attributes, potentially overlooking the synergistic effects between the instance and feature spaces, which could lead to distorted imputation outcomes. Furthermore, current methods uniformly treat all missing attributes as zero values, thus failing to differentiate between real and technical zeroes, potentially resulting in data over-imputation. To mitigate these challenges, we introduce a novel Reliable Attribute-Missing Multi-View Clustering method (RAM-MVC). Specifically, feature reconstruction is utilized to address missing attributes, while similarity graphs are simultaneously constructed within the instance and feature spaces. By leveraging structural information from both spaces, RAM-MVC learns a high-quality feature reconstruction matrix during the joint optimization process. Additionally, we introduce a reliable imputation guidance module that distinguishes between real and technical attribute-missing events, enabling discriminative imputation. The proposed RAM-MVC method outperforms nine baseline methods, as evidenced by real-world experiments using single-cell multi-view data.

Existing multiple kernel clustering (MKC) algorithms have two ubiquitous problems. From the theoretical perspective, most MKC algorithms lack sufficient theoretical analysis, especially the consistency of learned parameters, such as the kernel weights. From the practical perspective, the high complexity makes MKC unable to handle large-scale datasets. This paper tries to address the above two issues. We first make a consistency analysis of an influential MKC method named Simple Multiple Kernel$k$-Means (SimpleMKKM). Specifically, suppose that$\hat{\boldsymbol{\gamma }}_{n}$are the kernel weights learned by SimpleMKKM from the training samples. We also define the expected version of SimpleMKKM and denote its solution as$\boldsymbol{\gamma }^*$. We establish an upper bound of$\Vert \hat{\boldsymbol{\gamma }}_{n}-\boldsymbol{\gamma }^*\Vert _\infty$in the order of$\widetilde{\mathcal {O}}(1/\sqrt{n})$, where$n$is the sample number. Based on this result, we also derive its excess clustering risk calculated by a standard clustering loss function. For the large-scale extension, we replace the eigen decomposition of SimpleMKKM with singular value decomposition (SVD). Consequently, the complexity can be decreased to$\mathcal {O}(n)$such that SimpleMKKM can be implemented on large-scale datasets. We then deduce several theoretical results to verify the approximation ability of the proposed SVD-based method. The results of comprehensive experiments demonstrate the superiority of the proposed method. The code is publicly available athttps://github.com/weixuan-liang/SVD-based-SimpleMKKM.

Detecting 3D pedestrian from point cloud data in real-time while accounting for scale is crucial in various robotic and autonomous driving applications. Currently, the most successful methods for 3D object detection rely on voxel-based techniques, but these tend to be computationally inefficient for deployment in aerial scenarios. Conversely, the pillar-based approach exclusively employs 2D convolution, requiring fewer computational resources, albeit potentially sacrificing detection accuracy compared to voxel-based methods. Previous pillar-based approaches suffered from inadequate pillar feature encoding. In this letter, we introduce a real-time and scale-aware 3D Pedestrian Detection, which incorporates a robust encoder network designed for effective pillar feature extraction. The Proposed TriFocus Attention module (TriFA), which integrates external attention and similar attention strategies based on Squeeze and Exception. By comprehensively supervising the point-wise, channel-wise, and pillar-wise of pillar features, it enhances the encoding ability of pillars, suppresses noise in pillar features, and enhances the expression ability of pillar features. The proposed Bidirectional Scale-Aware Feature Pyramid module (BiSAFP) integrates a scale-aware module into the multi-scale pyramid structure. This addition enhances its ability to perceive pedestrian within low-level features. Moreover, it ensures that the significance of feature maps across various feature levels is fully taken into account. BiSAFP represents a lightweight multi-scale pyramid network that minimally impacts inference time while substantially boosting network performance. Our approach achieves real-time detection, processing up to 30 frames per second (FPS).

The multi-view hash method plays a pivotal role in multimedia retrieval, transforming diverse data from multiple perspectives into binary hash codes. While existing methods primarily emphasize complementarity across multiple views, they often face challenges associated with the non-convex nature of deep neural networks, ultimately causing a decrease in generalization ability. To overcome this limitation, we propose a novel curriculum calledAutomatic Multiple Loss Curriculum(AMLC). In AMLC, the deep multi-view hashing network undergoes training with data presented sequentially, progressing from simple to complex samples. This training strategy mirrors the human learning process, commencing with fundamental concepts and progressively advancing to tackle more intricate and abstract ideas. Building upon AMLC, we propose theBoosted Curriculum Multi-View Hashing(BCMVH) method. BCMVH facilitates the positioning of the learned model in a more flat region, enhancing its overall generalization capability. Extensive experiments conducted on three public datasets demonstrate that the proposed BCMVH outperforms state-of-the-art methods, achieving a maximum improvement of 3.17% in terms of mean Average Precision.

High-dimensional and complex spectral structures make the clustering of hyperspectral images (HSI) a challenging task. Subspace clustering is an effective approach for addressing this problem. However, current subspace clustering algorithms are primarily designed for a single view and do not fully exploit the spatial or textural feature information in HSI. In this study, contrastive multi-view subspace clustering of HSI was proposed based on graph convolutional networks. Pixel neighbor textural and spatial-spectral information were sent to construct two graph convolutional subspaces to learn their affinity matrices. To maximize the interaction between different views, a contrastive learning algorithm was introduced to promote the consistency of positive samples and assist the model in extracting robust features. An attention-based fusion module was used to adaptively integrate these affinity matrices, constructing a more discriminative affinity matrix. The model was evaluated using four popular HSI datasets: Indian Pines, Pavia University, Houston, and Xu Zhou. It achieved overall accuracies of 97.61%, 96.69%, 87.21%, and 97.65%, respectively, and significantly outperformed state-of-the-art clustering methods. In conclusion, the proposed model effectively improves the clustering accuracy of HSI. Our implementation is available at https://github.com/GuanRX/CMSCGC.

Hyperspectral image (HSI) clustering is a fundamental yet challenging task that groups image pixels with similar features into distinct clusters. Among various approaches, contrastive learning methods, which employ the concept of encouraging semantically similar samples to move closer together while pushing semantically inconsistent samples apart, have garnered significant attention due to their promising performance. However, the most prevalent approaches face two major limitations: 1) treating all samples indiscriminately during optimization, where the abundance of well-categorized samples overwhelms the feature learning process and 2) tending to introduce noise when constructing positive sample pairs through view augmentation or searching the nearest neighbors, which would cause semantic drift of sample features. To solve these issues, we propose a graph autoencoder-based deep clustering framework named spatial–spectral graph contrastive clustering with hard sample mining (SSGCC) that constructs spatial–spectral dual views without data augmentation and focuses more on hard samples rather than treating all samples equally with the aid of spatial–spectral features. Concretely, we extract the spectral features and the neighborhood spatial features of the samples as dual branches to avoid the noise caused by data augmentation and develop the cluster-oriented consistency learning to facilitate the exchange of knowledge between the two spectral–spatial perspectives. In addition, we propose a hard sample mining-based contrastive learning scheme with the aid of spatial–spectral features. To better measure the importance of the samples, we combine spatial features and spectral features to calculate the similarity between sample pairs. The weights of hard sample pairs are dynamically up-weight while the easy ones are down-weighting to improve the discriminative capability. Extensive experiments on four benchmark HSI datasets demonstrate the effectiveness and superiority of the proposed methods against state-of-the-art ones.

Fully supervised change detection methods have achieved significant advancements in performance, yet they depend severely on acquiring costly pixel-level labels. Considering that the patch-level annotations also contain abundant information corresponding to both changed and unchanged objects in bi-temporal images, an intuitive solution is to segment the changes with patch-level annotations. How to capture the semantic variations associated with the changed and unchanged regions from the patch-level annotations to obtain promising change results is the critical challenge for the weakly supervised change detection task. In this paper, we propose a memory-supported transformer (MS-Former), a novel framework consisting of a bi-directional attention block (BAB) and a patch-level supervision scheme (PSS) tailored for weakly supervised change detection with patch-level annotations. More specifically, the BAB captures contexts associated with the changed and unchanged regions from the temporal difference features to construct informative prototypes stored in the memory bank. On the other hand, the BAB extracts useful information from the prototypes as supplementary contexts to enhance the temporal difference features, thereby better distinguishing changed and unchanged regions. After that, the PSS guides the network learning valuable knowledge from the patch-level annotations, thus further elevating the performance. Experimental results on three benchmark datasets demonstrate the effectiveness of our proposed method in the change detection task. The demo code for our work will be publicly available at https://github.com/guanyuezhen/MS-Former.

Change detection (CD) aims to identify differences in the same scene at different times. With the increasing amount of hyperspectral images (HSIs), more and more change detection techniques use HSIs as the raw data. HSIs often contain redundant bands, where only a few are crucial for CD while others may be detrimental. However, most existing HSI-CD methods extract features directly from full-dimensional HSIs, leading to a degradation of feature discrimination. To tackle this issue, in this paper, we propose an end-to-end hyperspectral image change detection network based on band selection (ECDBS), unlocking the potential synergy between band selection and CD. The network compromises a deep learning based band selection module and cascaded band-specific spatial attention (BSA) blocks. The band selection module selectively retains bands favourable to CD according to the importance of the bands measured based on band correlation. The BSA block tailors the feature extraction strategy for each band based on its feature distribution, allowing extracting sufficient features from each band. Experimental evaluations were conducted on three widely used HSI-CD datasets, demonstrating the effectiveness and superiority of our proposed method over other state-of-the-art techniques.

In the realm of biomedicine, the prediction of associations between drugs and diseases holds significant importance. Yet, conventional wet lab experiments often fall short of meeting the stringent demands for prediction accuracy and efficiency. Many prior studies have predominantly focused on drug and disease similarities to predict drug-disease associations, but overlooking the crucial interactions between drugs and diseases that are essential for enhancing prediction accuracy. Hence, in this paper, a resilient and effective model named Hierarchical and Dynamic Graph Attention Network (HDGAT) has been proposed to predict drug-disease associations. Firstly, it establishes a heterogeneous graph by leveraging the interplay of drug and disease similarities and associations. Subsequently, it harnesses the capabilities of graph convolutional networks and bidirectional long short-term memory networks (Bi-LSTM) to aggregate node-level information within the heterogeneous graph comprehensively. Furthermore, it incorporates a hierarchical attention mechanism between convolutional layers and a dynamic attention mechanism between nodes to learn embeddings for drugs and diseases. The hierarchical attention mechanism assigns varying weights to embeddings learned from different convolutional layers, and the dynamic attention mechanism efficiently prioritizes inter-node information by allocating each node with varying rankings of attention coefficients for neighbour nodes. Moreover, it employs residual connections to alleviate the over-smoothing issue in graph convolution operations. The latent drug-disease associations are quantified through the fusion of these embeddings ultimately. By conducting 5-fold cross-validation, HDGAT's performance surpasses the performance of existing state-of-the-art models across various evaluation metrics, which substantiates the exceptional efficacy of HDGAT in predicting drug-disease associations.

Medication recommendation is a productive task for AI-driven healthcare systems, which can assist clinicians in prescribing judicious and effective treatments. However, existing medication recommendation methods omit two key pieces of information: Coarse-grained interaction information between distinct types of symptoms in a patient's medical history and corresponding medication representations can serve as attention for predicting the current medication combinations of the patient. Fine-grained interaction information between medication substructure representations and different types of symptoms can facilitate the construction of molecular-level disentangled medication representations. To address this dilemma, we propose a novelDualAdaptiveInteractionNetwork (DAI-Net), which encodes comprehensive interaction knowledge between patients' multifaceted health records and medication molecules to improve the performance of medication recommendation and heighten interpretability of the model. Specifically, we design a symptom-aware medication matching module to extract coordinated associations between patient symptoms and medication molecules, coarse-grained interaction learning. The medication embeddings are utilized to transform patient-medication matching properties into a symptom-substructure matching matrix for fine-grained interaction. The patient's Longitudinal representation is employed as a query to decode both symptom-medication and symptom-substructure matching information for coordinated medication representation. DAI-Net is an end-to-end recommendation model. Extensive experiments on the real-world EHR datasets, i.e., the public benchmark MIMIC-III, MIMIC-IV, and eICU, demonstrate that the proposed DAI-Net achieves competitive performance compared to other state-of-the-art ones, with an average improvement of 1.8%, 2.1% in Jaccard on MIMIC-III and -IV dataset.

Multiple kernel clustering (MKC) enhances clustering performance by deriving a consensus partition or graph from a predefined set of kernels. Despite many advanced MKC methods proposed in recent years, the prevalent approaches involve incorporating all kernels by default to capture diverse information within the data. However, learning from all kernels may not be better than one of a few kernels, particularly since some kernels exhibit a higher proportion of noise than semantic content. Additionally, existing MKC methods, whether based on early-fusion or late-fusion approaches, predominantly rely on pairwise relationships among samples or cluster structures, neglecting potential correlations between these two aspects. To this end, we propose a multiple kernel clustering with an adaptive multi-scale partition selection method (MPS), which exploits multiple-dimensional representations and the pairwise cluster structure for clustering. By the proposed kernel selection framework, potentially harmful kernels are dynamically excluded during the kernel fusion process, and then the multi-scale partitions and similarity graphs derived from the retained kernels are utilized to facilitate the improved consensus partition generation. Finally, extensive experiments are conducted to demonstrate the effectiveness of MPS on eight benchmark datasets.

Multiple Kernel Clustering (MKC) has emerged as a prominent research domain in recent decades due to its capacity to exploit diverse information from multiple views by learning an optimal kernel. Despite the successes achieved by various MKC methods, a significant challenge lies in the computational complexity associated with generating a consensus partition from the optimal kernel matrix, typically of size$n \times n$, where$n$represents the number of samples. This computational bottleneck restricts the practical applicability of these methods when confronted with large-scale datasets. Furthermore, certain existing MKC algorithms derive the consensus partition matrix by fusing all base partitions. However, this fusion process may inadvertently overlook critical information embedded in individual base kernels, potentially leading to inferior clustering performance. In light of these challenges, we introduce an innovative and efficient multiple kernel$k$-means approach, denoted as FAMKKM. Notably, FAMKKM incorporates two approximated partition matrices instead of the original individual partition matric for each base kernel. This strategic substitution significantly reduces computational complexity. Additionally, FAMKKM leverages the original kernel information to guide the fusion of all base partitions, thereby enhancing the quality of the resulting consensus partition matrix. Finally, we substantiate the efficacy and efficiency of the proposed FAMKKM through extensive experiments conducted on six benchmark datasets. Our results demonstrate its superiority over state-of-the-art methods. The demo code of this work is publicly available athttps://github.com/WangJun2023/FAMKKM

Multiple kernel clustering (MKC) aims to extract and integrate the clustering information from a set of pre-defined kernels for handling data which cannot be linearly separated well. More precisely, existing MKC methods generally devote to learn the complementary information from a set of kernel partitions, whose feature dimensions are commonly fixed as the upper bound$n$or lower bound$c$, where$n$and$c$represents the number of samples and clusters, respectively. However, the adopting of the lower bound or upper bound generally leads to poor clustering performance caused by the lack or redundancy of clustering information carried by kernel partitions. To tackle this issue, we propose a novel late fusion multiple kernel clustering method, termed as Eigenvalue Ratio Inspired Partition Learning and Fusion for Multiple Kernel Clustering (ERMKC), in this paper. Specifically, we propose an eigenvalue ratio based criterion to guide the kernel partition learning for each single kernel matrix, which ensures more suitable feature dimensions for the learnt kernel partitions. In addition, we also propose a novel late fusion model for fusing the learnt kernel partitions optimally. Furthermore, we conduct extensive experiments on numerous benchmark datasets to evaluate the proposed ERMKC method, whose results verify the effectiveness and advantage of the proposed method compared to the other state-of-the-art methods.

Spatial transcriptomics technology fully leverages spatial location and gene expression information for spatial clustering tasks. However, existing spatial clustering methods primarily concentrate on utilizing the complementary features between spatial and gene expression information, while overlooking the discriminative features during the integration process. Consequently, the discriminative capability of node representation in the gene expression features is limited. Besides, most existing methods lack a flexible combination mechanism to adaptively integrate spatial and gene expression information. To this end, we propose an end-to-end deep learning method named MAFN for spatially resolved transcriptomics data clustering via a multi-view adaptive fusion network. Specifically, we first adaptively learn inter-view complementary features from spatial and gene expression information. To improve the discriminative capability of gene expression nodes by utilizing spatial information, we employ two GCN encoders to learn intra-view specific features and design a Cross-view Correlation Reduction (CCR) strategy to filter the irrelevant information. Moreover, considering the distinct characteristics of each view, a Cross-view Attention Module (CAM) is utilized to adaptively fuse the multi-view features. Extensive experimental results demonstrate that the proposed MAFN achieves competitive performance in spatial domain identification compared to other state-of-the-art ones.

Multiview clustering algorithms have attracted intensive attention and achieved superior performance in various fields recently. Despite the great success of multiview clustering methods in realistic applications, we observe that most of them are difficult to apply to large-scale datasets due to their cubic complexity. Moreover, they usually use a two-stage scheme to obtain the discrete clustering labels, which inevitably causes a suboptimal solution. In light of this, an efficient and effective one-step multiview clustering (E2OMVC) method is proposed to directly obtain clustering indicators with a small-time burden. Specifically, according to the anchor graphs, the smaller similarity graph of each view is constructed, from which the low-dimensional latent features are generated to form the latent partition representation. By introducing a label discretization mechanism, the binary indicator matrix can be directly obtained from the unified partition representation which is formed by fusing all latent partition representations from different views. In addition, by coupling the fusion of all latent information and the clustering task into a joint framework, the two processes can help each other and obtain a better clustering result. Extensive experimental results demonstrate that the proposed method can achieve comparable or better performance than the state-of-the-art methods. The demo code of this work is publicly available at https://github.com/WangJun2023/EEOMVC.

With the rapid development of various data acquisition technologies, more and more multimodal data come into being. It is important to integrate different modalities which are with high-dimensional features for boosting final multimodal data classification task. However, existing multimodal classification methods mainly focus on exploiting the complementary information of different modalities, while ignoring the learning confidence during information fusion. In this paper, we propose a trustworthy multimodal classification network via multi-level confidence learning, referred to as MLCLNet. Considering that a large number of feature dimensions could not contribute to final classification performance but disturb the discriminability of different samples, we propose a feature confidence learning mechanism to suppress some redundant features, as well as enhancing the expression of discriminative feature dimensions in each modality. In order to capture the inherent sample structure information implied in each modality, we design a graph convolutional network branch to learn the corresponding structure preserved feature representation and generate modal-specific initial classification labels. Since samples from different modalities should share consistent labels, a cross-modal label fusion module is deployed to capture the label correlations of different modalities. In addition, motivated the ideally orthogonality of final fused label matrix, we design a label confidence loss to supervise the network for learning more separable data representations. To the best of our knowledge, MLCLNet is the first work which integrates both feature and label-level confidence learning for multimodal classification. Extensive experiments on four multimodal medical datasets are conducted to validate superior performance of MLCLNet when compared to other state-of-the-art methods.

Multi-view clustering can partition data samples into their categories by learning a consensus representation in unsupervised way and has received more and more attention in recent years. However, most existing deep clustering methods learn consensus representation or view-specific representations from multiple views via view-wise aggregation way, where they ignore structure relationship of all samples. In this paper, we propose a novel multi-view clustering network to address these problems, called Global and Cross-view Feature Aggregation for Multi-View Clustering (GCFAggMVC). Specifically, the consensus data presentation from multiple views is obtained via cross-sample and cross-view feature aggregation, which fully explores the complementary of similar samples. Moreover, we align the consensus representation and the view-specific representation by the structure-guided contrastive learning module, which makes the view-specific representations from different samples with high structure relationship similar. The proposed module is a flexible multi-view data representation module, which can be also embedded to the incomplete multi-view data clustering task via plugging our module into other frameworks. Extensive experiments show that the proposed method achieves excellent performance in both complete multi-view data clustering tasks and incomplete multi-view data clustering tasks.

Multimodal learning aims to integrate complementary information from different modalities for more reliable decisions. However, existing multimodal classification methods simply integrate the learned local features, which ignore the underlying structure of each modality and the higher-order correlation across modalities. In this paper, we propose a novel Hierarchical Attention Learning Network (HALNet) for multimodal classification. Specifically, HALNet has three merits: 1) A hierarchical feature fusion module is proposed to learn multilevel features, aggregating multi-level features for a global feature representation with the attention mechanism and progressive fusion tactics. 2) A cross-modal higher-order fusion module is introduced to capture the prospective cross-modal correlations at label space. 3) A dual prediction pattern is designed to generate credible decisions. Extensive experiments on three real-world multimodal datasets demonstrate that HALNet achieves competitive performance compared to the state-of-the-art.

Multispectral object detection has gained significant attention due to its potential in all-weather applications, particularly those involving visible (RGB) and infrared (IR) images. Despite substantial advancements in this domain, current methodologies primarily rely on rudimentary accumulation operations to combine complementary information from disparate modalities, overlooking the semantic conflicts that arise from the intrinsic heterogeneity among modalities. To address this issue, we propose a novel learning network, the Cross-modal Conflict-Aware Learning Network (CALNet), that takes into account semantic conflicts and complementary information within multi-modal input. Our network comprises two pivotal modules: the Cross-Modal Conflict Rectification Module (CCR) and the Selected Cross-modal Fusion (SCF) Module. The CCR module mitigates modal heterogeneity by examining contextual information of analogous pixels, thus alleviating multi-modal information with semantic conflicts. Subsequently, semantically coherent information is supplied to the SCF module, which fuses multi-modal features by assessing intra-modal importance to select semantically rich features and mining inter-modal complementary information. To assess the effectiveness of our proposed method, we develop a two-stream one-stage detector based on CALNet for multispectral object detection. Comprehensive experimental outcomes demonstrate that our approach considerably outperforms existing methods in resolving the cross-modal semantic conflict issue and achieving state-of-the-art accuracy in detection results.

With advances in sensing technology, multi-modal data collected from different sources are increasingly available. Multi-modal classification aims to integrate complementary information from multi-modal data to improve model classification performance. However, existing multi-modal classification methods are basically weak in integrating global structural information and providing trustworthy multi-modal fusion, especially in safety-sensitive practical applications (e.g., medical diagnosis). In this paper, we propose a novel Dynamic Poly-attention Network (DPNET) for trustworthy multi-modal classification. Specifically, DPNET has four merits: (i) To capture the intrinsic modality-specific structural information, we design a structure-aware feature aggregation module to learn the corresponding structure-preserved global compact feature representation. (ii) A transparent fusion strategy based on the modality confidence estimation strategy is induced to track information variation within different modalities for dynamical fusion. (iii) To facilitate more effective and efficient multi-modal fusion, we introduce a cross-modal low-rank fusion module to reduce the complexity of tensor-based fusion and activate the implication of different rank-wise features via a rank attention mechanism. (iv) A label confidence estimation module is devised to drive the network to generate more credible confidence. An intra-class attention loss is introduced to supervise the network training. Extensive experiments on four real-world multi-modal biomedical datasets demonstrate that the proposed method achieves competitive performance compared to other state-of-the-art ones.

Hyperspectral band selection aims to identify an optimal subset of bands for hyperspectral images (HSIs). For most existing clustering-based band selection methods, they directly stretch each band into a single feature vector and employ the pixelwise features to address band redundancy. In this way, they do not take full consideration of the spatial information and deal with the importance of different regions in HSIs, which leads to a nonoptimal selection. To address these issues, a region-aware hierarchical latent feature representation learning-guided clustering (HLFC) method is proposed. Specifically, in order to fully preserve the spatial information of HSIs, the superpixel segmentation algorithm is adopted to segment HSIs into multiple regions first. For each segmented region, the similarity graph is constructed to reflect the bands-wise similarity, and its corresponding Laplacian matrix is generated for learning low-dimensional latent features in a hierarchical way. All latent features are then fused to form a unified feature representation of HSIs. Finally, k -means clustering is utilized on the unified feature representation matrix to generate multiple clusters from which the band with maximum information entropy is selected to form the final subset of bands. Extensive experimental results demonstrate that the proposed clustering method can achieve superior performance than the state-of-the-art representative methods on the band selection. The demo code of this work is publicly available at https://github.com/WangJun2023/HLFC.

Deep learning-based hyperspectral image (HSI) classification and object detection techniques have gained significant attention due to their vital role in image content analysis, interpretation, and broader HSI applications. However, current hyperspectral object detection approaches predominantly emphasize spectral or spatial information, overlooking the valuable complementary relationship between these two aspects. In this study, we present a novel Spectral-Spatial Aggregation (S2ADet) object detector that effectively harnesses the rich spectral and spatial complementary information inherent in the hyperspectral image. S2ADet comprises a hyperspectral information decoupling (HID) module, a two-stream feature extraction network, and a one-stage detection head. The HID module processes hyperspectral data by aggregating spectral and spatial information via band selection and principal components analysis, consequently reducing redundancy. Based on the acquired spectral and spatial aggregation information, we propose a feature aggregation two-stream network for interacting spectral-spatial features. Furthermore, to address the limitations of existing databases, we annotate an extensive dataset, designated as HOD3K, containing 3,242 hyperspectral images captured across diverse real-world scenes and encompassing three object classes. These images possess a resolution of 512×256 pixels and cover 16 bands ranging from 470 nm to 620 nm. Comprehensive experiments on two datasets demonstrate that S2ADet surpasses existing state-of-the-art methods, achieving robust and reliable results. The demo code and dataset of this work are publicly available at https://github.com/hexiao-cs/S2ADet.

Remote sensing change detection (RSCD) aims to explore surface changes from co-registered pair of images. However, the high cost of memory and computation in previous convolutional neural network (CNN)-based methods prevent their successes from being applied to real-world applications. Therefore, we propose a novel lightweight network, which identifies changes based on the features extracted by mobile networks via progressive feature aggregation and supervised attention, termed as A2Net. Considering the less powerful representation capability of mobile networks, we design a neighbor aggregation module (NAM) to fuse features within nearby stages of the backbone to strengthen the representation capability of temporal features. Then, we propose a progressive change identifying module (PCIM) to extract temporal difference information from bitemporal features. Besides, we design a supervised attention module (SAM) to reweight features for effectively aggregating multilevel features from high levels to low levels. With NAM, PCIM, and SAM incorporated, A2Net can achieve favorable results compared with the state-of-the-art methods on three challenging RSCD datasets with fewer parameters (3.78 M) and lower computation costs (6.02 G). The demo code of this work is publicly available athttps://github.com/guanyuezhen/A2Net.

As an effective manner to reduce data redundancy and processing inconvenience, hyperspectral band selection aims to select a subset of informative and discriminative bands from the original data cube. Although a large number of approaches have been proposed and obtained great success, they still face at least two issues. Firstly, most of the previous methods only consider the redundancy between neighbor bands, while the global information has been ignored. Secondly, each band is often treated as a whole and reshaped to a feature vector without considering the spatial structure of different regions. In this paper, in order to address these issues, we propose a spatial and spectral structure preserved self-representation model for unsupervised hyperspectral band selection without using any label information, referred to as S4P briefly. Different from previous methods that stretch each band into a feature vector, the first principal component of the original hyperspectral cube is segmented into different superpixels, which can reflect the spatial structure of homogeneous regions. Then each band can be represented by a superpixel level feature vector and the self-representation model is utilized to learn the spectral correlation of different bands. In addition, an adaptive and weighted multiple graph fusion term is designed to generate a unified similarity graph between different superpixels, which is used to capture the spatial structure in the self-representation space. Finally, anl2,1-norm is imposed on the self-representation coefficient matrix to measure the band importance. We design an alternative update scheme to optimize the resultant problem, the self-representation coefficient matrix and the superpixel-wise similarity graph can boost each other during the updating process to obtain optimal results. Extensive experiments with detailed analysis of three public datasets are conducted to validate the superiority of the proposed S4P when compared with other state-of-the-art competitors.

Multi-view spectral clustering, which exploits the complementary information among graphs of diverse views to obtain superior clustering results, has attracted intensive attention recently. However, most existing multi-view spectral clustering methods obtain the clustering partitions in a two-step scheme, i.e., spectral embedding and subsequent$k$-means. This two-step scheme inevitably seeks sub-optimal clustering results due to the information loss during the two-steps processes. Besides, existing multi-view spectral clustering methods do not jointly utilize the information of graphs and embedding matrices, which also degrades final clustering results. To solve these issues, we propose a unified one-step multi-view spectral clustering method, which integrates the spectral embedding and$k$-means into a unified framework to obtain discrete clustering labels with a one-step strategy. Under the observation that the inner product of the embedding matrix is a low-rank approximation of the graph, we combine graphs and embedding matrices of different views to obtain a unified graph. Then, we directly capture the discrete clustering indicator matrix from the unified graph. Furthermore, we design an effective optimization algorithm to solve the resultant problem. Finally, a set of experiments on various datasets are conducted to verify the effectiveness of the proposed method. The demo code of this work is publicly available atrgb]0,0,1https://github.com/guanyuezhen/UOMvSC.

Clustering is a fundamental task in the machine learning and data mining community. Among existing clustering methods, multiple kernel clustering (MKC) has been widely investigated due to its effectiveness to capture non-linear relationships among samples. However, most of the existing MKC methods bear intensive computational complexity in learning an optimal kernel and seeking the final clustering partition. In this paper, based on the spectral perturbation theory, we propose an efficient MKC method that reduces the computational complexity from O(n3) to O(nk2 + k3), with n and k denoting the number of data samples and the number of clusters, respectively. The proposed method recovers the optimal clustering partition from base partitions by maximizing the eigen gaps to approximate the perturbation errors. An equivalent optimization objective function is introduced to obtain base partitions. Furthermore, a kernel weighting scheme is embedded to capture the diversity among multiple kernels. Finally, the optimal partition, base partitions, and kernel weights are jointly learned in a unified framework. An efficient alternate iterative optimization algorithm is designed to solve the resultant optimization problem. Experimental results on various benchmark datasets demonstrate the superiority of the proposed method when compared to other state-of-the-art ones in terms of both clustering efficacy and efficiency.

Multi-view clustering is an important technique in machine learning research. Existing methods have improved in clustering performance, most of them learn graph structure depending on all samples, which are high complexity. Bipartite graph-based multi-view clustering can obtain clustering result by establishing the relationship between the sample points and small anchor points, which improve the efficiency of clustering. Most bipartite graph-based clustering methods only focus on topological graph structure learning depending on sample nodes, ignore the influence of node features. In this paper, we propose bipartite graph-based discriminative feature learning for multi-view clustering, which combines bipartite graph learning and discriminative feature learning to a unified framework. Specifically, the bipartite graph learning is proposed via multi-view subspace representation with manifold regularization terms. Meanwhile, our feature learning utilizes data pseudo-labels obtained by fused bipartite graph to seek projection direction, which make the same label be closer and make data points with different labels be far away from each other. At last, the proposed manifold regularization terms establish the relationship between constructed bipartite graph and new data representation. By leveraging the interactions between structure learning and discriminative feature learning, we are able to select more informative features and capture more accurate structure of data for clustering. Extensive experimental results on different scale datasets demonstrate our method achieves better or comparable clustering performance than the results of state-of-the-art methods.

Albeit great success has been achieved in image defocus blur detection, there are still several unsolved challenges, e.g., interference of background clutter, scale sensitivity and missing boundary details of blur regions. To deal with these issues, we propose a deep neural network which recurrently fuses and refines multi-scale deep features (DeFusionNet) for defocus blur detection. We first fuse the features from different layers of FCN as shallow features and semantic features, respectively. Then, the fused shallow features are propagated to deep layers for refining the details of detected defocus blur regions, and the fused semantic features are propagated to shallow layers to assist in better locating blur regions. The fusion and refinement are carried out recurrently. In order to narrow the gap between low-level and high-level features, we embed a feature adaptation module before feature propagating to exploit the complementary information as well as reduce the contradictory response of different feature layers. Since different feature channels are with different extents of discrimination for detecting blur regions, we design a channel attention module to select discriminative features for feature refinement. Finally, the output of each layer at last recurrent step are fused to obtain the final result. We collect a new dataset consists of various challenging images and their pixel-wise annotations for promoting further study. Extensive experiments on two commonly used datasets and our newly collected one are conducted to demonstrate both the efficacy and efficiency of DeFusionNet.

Remote sensing change detection (RSCD), which identifies the changed and unchanged pixels from a registered pair of remote sensing images, has enjoyed remarkable success recently. However, locating changed objects with fine structural details is still a challenging problem in RSCD. In this paper, we propose a novel remote sensing change detection network via temporal feature interaction and guided refinement (TFI-GR) to solve this issue. Specifically, unlike previous methods, which just employ one single concatenation or subtraction operation for bi-temporal feature fusion, we design a temporal feature interaction module (TFIM) to enhance interaction between bi-temporal features and capture temporal difference information at diverse feature levels. Afterword, a guided refinement modules (GRM), which aggregates both low- and high-level temporal difference representations to polish the location information of high-level features and filter the background clutters of low-level features, is repeatedly performed. Finally, the multi-level temporal difference features are progressively fused to generate change maps for change detection. To demonstrate the effectiveness of the proposed TFI-GR, comprehensive experiments are performed on three high spatial resolution remote sensing change detection datasets. Experimental results indicate that the proposed method is superior to other state-of-the-art change detection methods. The demo code of this work is publicly available at https://github.com/guanyuezhen/TFI-GR.

Incomplete multi-view clustering aims to exploit the information of multiple incomplete views to partition data into their clusters. Existing methods only utilize the pair-wise sample correlation and pair-wise view correlation to improve the clustering performance but neglect the high-order correlation of samples and that of views. To address this issue, we propose a high-order correlation preserved incomplete multi-view subspace clustering (HCP-IMSC) method which effectively recovers the missing views of samples and the subspace structure of incomplete multi-view data. Specifically, multiple affinity matrices constructed from the incomplete multi-view data are treated as a third-order low rank tensor with a tensor factorization regularization which preserves the high-order view correlation and sample correlation. Then, a unified affinity matrix can be obtained by fusing the view-specific affinity matrices in a self-weighted manner. A hypergraph is further constructed from the unified affinity matrix to preserve the high-order geometrical structure of the data with incomplete views. Then, the samples with missing views are restricted to be reconstructed by their neighbor samples under the hypergraph-induced hyper-Laplacian regularization. Furthermore, the learning of view-specific affinity matrices as well as the unified one, tensor factorization, and hyper-Laplacian regularization are integrated into a unified optimization framework. An iterative algorithm is designed to solve the resultant model. Experimental results on various benchmark datasets indicate the superiority of the proposed method. The code is implemented by using MATLAB R2018a and MindSpore library: https://github.com/ChangTang/HCP-IMSC.

Although demonstrating great success, previous multi-view unsupervised feature selection (MV-UFS) methods often construct a view-specific similarity graph and characterize the local structure of data within each single view. In such a way, the cross-view information could be ignored. In addition, they usually assume that different feature views are projected from a latent feature space while the diversity of different views cannot be fully captured. In this work, we resent a MV-UFS model via cross-view local structure preserved diversity and consensus learning, referred to as CvLP-DCL briefly. In order to exploit both the shared and distinguishing information across different views, we project each view into a label space, which consists of a consensus part and a view-specific part. Therefore, we regularize the fact that different views represent same samples. Meanwhile, a cross-view similarity graph learning term with matrix-induced regularization is embedded to preserve the local structure of data in the label space. By imposing the$l_{2,1}$-norm on the feature projection matrices for constraining row sparsity, discriminative features can be selected from different views. An efficient algorithm is designed to solve the resultant optimization problem and extensive experiments on six publicly datasets are conducted to validate the effectiveness of the proposed CvLP-DCL.

Multi-view clustering, which exploits the multi-view information to partition data into their clusters, has attracted intense attention. However, most existing methods directly learn a similarity graph from original multi-view features, which inevitably contain noises and redundancy information. The learned similarity graph is inaccurate and is insufficient to depict the underlying cluster structure of multi-view data. To address this issue, we propose a novel multi-view clustering method that is able to construct an essential similarity graph in a spectral embedding space instead of the original feature space. Concretely, we first obtain multiple spectral embedding matrices from the view-specific similarity graphs, and reorganize the gram matrices constructed by the inner product of the normalized spectral embedding matrices into a tensor. Then, we impose a weighted tensor nuclear norm constraint on the tensor to capture high-order consistent information among multiple views. Furthermore, we unify the spectral embedding and low rank tensor learning into a unified optimization framework to determine the spectral embedding matrices and tensor representation jointly. Finally, we obtain the consensus similarity graph from the gram matrices via an adaptive neighbor manner. An efficient optimization algorithm is designed to solve the resultant optimization problem. Extensive experiments on six benchmark datasets are conducted to verify the efficacy of the proposed method. The code is implemented by using MATLAB R2018a and MindSpore library[1]:https://github.com/guanyuezhen/CGL.

As a representative of multiple kernel clustering (MKC), simple multiple kernel k-means (SimpleMKKM) is recently put forward to boosting the clustering performance by optimally fusing a group of pre-specified kernel matrices. Despite achieving significant improvement in a variety of applications, we find out that SimpleMKKM could indiscriminately force all sample pairs to be equally aligned with the same ideal similarity. As a result, it does not sufficiently take the variation of samples into consideration, leading to unsatisfying clustering performance. To address these issues, this paper proposes a novel MKC algorithm with a "local" kernel alignment, which only requires that the similarity of a sample to its k-nearest neighbours be aligned with the ideal similarity matrix. Such an alignment helps the clustering algorithm to focus on closer sample pairs that shall stay together and avoids involving unreliable similarity evaluation for farther sample pairs. After that, we theoretically show that the objective of SimpleMKKM is a special case of this local kernel alignment criterion with normalizing each base kernel matrix. Based on this observation, the proposed localized SimpleMKKM can be readily implemented by existing SimpleMKKM package. Moreover, we conduct extensive experiments on several widely used benchmark datasets to evaluate the clustering performance of localized SimpleMKKM. The experimental results have demonstrated that our algorithm consistently outperforms the state-of-the-art ones, verifying the effectiveness of the proposed local kernel alignment criterion. The code of Localized SimpleMKKM is publicly available at: https://github.com/xinwangliu/LocalizedSMKKM.

In this paper, we propose a novel incomplete multi-view clustering method, in which a tensor nuclear norm regularizer elegantly diffuses the information of multi-view block-diagonal structure across different views. By exploring the membership between observed and missing samples and that between missing ones in each incomplete view with the guidance of the high-order view consistency, a global block-diagonal structure is well preserved in multiple spectral embedding matrices. Meanwhile, a consensus representation with strong separability is obtained for clustering. An iterative algorithm based on Augmented Lagrange Multiplier (ALM) is designed to solve the resultant model. Experimental results on six benchmark datasets indicate the superiority of the proposed method. http://github.com/ChangTang/TMBSD

Existing late fusion multi-view clustering (LFMVC) optimally integrates a group of pre-specified base partition matrices to learn a consensus one. It is then taken as the input of the widely used k-means to generate the cluster labels. As observed, the learning of the consensus partition matrix and the generation of cluster labels are separately done. These two procedures lack necessary negotiation and can not best serve for each other, which may adversely affect the clustering performance. To address this issue, we propose to unify the aforementioned two learning procedures into a single optimization, in which the consensus partition matrix can better serve for the generation of cluster labels, and the latter is able to guide the learning of the former. To optimize the resultant optimization problem, we develop a four-step alternate algorithm with proved convergence. We theoretically analyze the clustering generalization error of the proposed algorithm on unseen data. Comprehensive experiments on multiple benchmark datasets demonstrate the superiority of our algorithm in terms of both clustering accuracy and computational efficiency. It is expected that the simplicity and effectiveness of our algorithm will make it a good option to be considered for practical multi-view clustering applications.

In this paper, we propose a hyperspectral band selection method via spatial-spectral weighted region-wise multiple graph fusion-based spectral clustering, referred to as RMGF briefly. Considering that different objects have different reflection characteristics, we use a superpixel segmentation algorithm to segment the first principal component of original hyperspectral image cube into homogeneous regions. For each superpixel, we construct a corresponding similarity graph to reflect the similarity between band pairs. Then, a multiple graph diffusion strategy with theoretical convergence guarantee is designed to learn a unified graph for partitioning the whole hyperspectral cube into several subcubes via spectral clustering. During the graph diffusion process, the spatial and spectral information of each superpixel are embedded to make spatial/spectral similar superpixels contribute more to each other. Finally, the band containing minimum noise in each subcube is selected to represent the whole subcube. Extensive experiments are conducted on three public datasets to validate the superiority of the proposed method when compared with other state-of-the-art ones.

Incomplete multi-view clustering (IMVC) optimally combines multiple pre-specified incomplete views to improve clustering performance. Among various excellent solutions, the recently proposed multiple kernel k-means with incomplete kernels (MKKM-IK) forms a benchmark, which redefines IMVC as a joint optimization problem where the clustering and kernel matrix imputation tasks are alternately performed until convergence. Though demonstrating promising performance in various applications, we observe that the manner of kernel matrix imputation in MKKM-IK would incur intensive computational and storage complexities, over-complicated optimization and limitedly improved clustering performance. In this paper, we first propose an Efficient and Effective Incomplete Multi-view Clustering (EE-IMVC) algorithm to address these issues. Instead of completing the incomplete kernel matrices, EE-IMVC proposes to impute each incomplete base matrix generated by incomplete views with a learned consensus clustering matrix. Moreover, we further improve this algorithm by incorporating prior knowledge to regularize the learned consensus clustering matrix. Two three-step iterative algorithms are carefully developed to solve the resultant optimization problems with linear computational complexity, and their convergence is theoretically proven. After that, we theoretically study the generalization bound of the proposed algorithms. Furthermore, we conduct comprehensive experiments to study the proposed algorithms in terms of clustering accuracy, evolution of the learned consensus clustering matrix and the convergence. As indicated, our algorithms deliver their effectiveness by significantly and consistently outperforming some state-of-the-art ones.

Simultaneous clustering and optimization (SCO) has recently drawn much attention due to its wide range of practical applications. Many methods have been previously proposed to solve this problem and obtain the optimal model. However, when a dataset evolves over time, those existing methods have to update the model frequently to guarantee accuracy; such updating is computationally infeasible. In this paper, we propose a new formulation of SCO to handle evolving datasets. Specifically, we propose a new variant of the alternating direction method of multipliers (ADMM) to solve this problem efficiently. The guarantee of model accuracy is analyzed theoretically for two specific tasks: ridge regression and convex clustering. Extensive empirical studies confirm the effectiveness of our method.

Due to the remarkable potential applications, defocus blur detection, which aims to separate blurry regions from an image, has attracted much attention. Although significant progress has been made by many methods, there are still various challenges that hinder the results, e.g., confusing background areas, sensitivity to the scale and missing the boundary details of the defocus blur regions. To solve these issues, in this paper, we propose a deep convolutional neural network (CNN) for defocus blur detection via a Bi-directional Residual Refining network (BR2Net). Specifically, a residual learning and refining module (RLRM) is designed to correct the prediction errors in the intermediate defocus blur map. Then, we develop a bidirectional residual feature refining network with two branches by embedding multiple RLRMs into it for recurrently combining and refining the residual features. One branch of the network refines the residual features from the shallow layers to the deep layers, and the other branch refines the residual features from the deep layers to the shallow layers. In such a manner, both the low-level spatial details and highlevel semantic information can be encoded step by step in two directions to suppress background clutter and enhance the detected region details. The outputs of the two branches are fused to generate the final results. In addition, with the observation that different feature channels have different extents of discrimination for detecting blurred regions, we add a channel attention module to each feature extraction layer to select more discriminative features for residual learning. To promote further research on defocus blur detection, we create a new dataset with various challenging images and manually annotate their corresponding pixelwise ground truths. The proposed network is validated on two commonly used defocus blur detection datasets and our newly collected dataset by comparing it with 10 other state-of-the-art methods. Extensive experiments with ablation studies demonstrate that BR2Net consistently and significantly outperforms the competitors in terms of both the efficiency and accuracy.

Graph based multi-view clustering has been paid great attention by exploring the neighborhood relationship among data points from multiple views. Though achieving great success in various applications, we observe that most of previous methods learn a consensus graph by building certain data representation models, which at least bears the following drawbacks. First, their clustering performance highly depends on the data representation capability of the model. Second, solving these resultant optimization models usually results in high computational complexity. Third, there are often some hyper-parameters in these models need to tune for obtaining the optimal results. In this work, we propose a general, effective and parameter-free method with convergence guarantee to learn a unified graph for multi-view data clustering via cross-view graph diffusion (CGD), which is the first attempt to employ diffusion process for multi-view clustering. The proposed CGD takes the traditional predefined graph matrices of different views as input, and learns an improved graph for each single view via an iterative cross diffusion process by 1) capturing the underlying manifold geometry structure of original data points, and 2) leveraging the complementary information among multiple graphs. The final unified graph used for clustering is obtained by averaging the improved view associated graphs. Extensive experiments on several benchmark datasets are conducted to demonstrate the effectiveness of the proposed method in terms of seven clustering evaluation metrics.

Defocus blur detection aims to separate the in-focus and out-of-focus regions in an image. Although attracting more and more attention due to its remarkable potential applications, there are still several challenges for accurate defocus blur detection, such as the interference of background clutter, sensitivity to scales and missing boundary details of defocus blur regions. In order to address these issues, we propose a deep neural network which Recurrently Refines Multi-scale Residual Features (R2MRF) for defocus blur detection. We firstly extract multi-scale deep features by utilizing a fully convolutional network. For each layer, we design a novel recurrent residual refinement branch embedded with multiple residual refinement modules (RRMs) to more accurately detect blur regions from the input image. Considering that the features from bottom layers are able to capture rich low-level features for details preservation while the features from top layers are capable of characterizing the semantic information for locating blur regions, we aggregate the deep features from different layers to learn the residual between the intermediate prediction and the ground truth for each recurrent step in each residual refinement branch. Since the defocus degree is sensitive to image scales, we finally fuse the side output of each branch to obtain the final blur detection map. We evaluate the proposed network on two commonly used defocus blur detection benchmark datasets by comparing it with other 11 state-of-the-art methods. Extensive experimental results with ablation studies demonstrate that R2MRF consistently and significantly outperforms the competitors in terms of both efficiency and accuracy.

With the growing of chronic diseases, strain on public healthcare systems becomes a critical problem of our society. Internet of Things (IoT) technology [1]–[3], as a hot topic, has attracted more and more attention recently due to its potential ability to reduce the strain on public healthcare systems. In IoT based healthcare system, the information of patients can be automatically obtained via various sensors and the physical condition of patients can be analyzed via those obtained data. In this work, we present a IoT and blockchain based healthcare system for human activity recognition via monitoring vital/non-vital signals collected from wearable-sensors remotely. On the one hand, the IoT technology is used for data acquisition and transmission [4]. On the other hand, the blockchain technology is used for data encryption. An incremental learning strategy based on covariance matrix is designed to recognize the activity done by a patient under care and determine whether there is an emergency for the patient under care. In such a manner, necessary feedback or programmable alarms can be made during or after the activity.

Feature extraction and feature selection have been regarded as two independent dimensionality reduction methods in most of the existing literature. In this paper, we propose to integrate both approaches into a unified framework and design an unsupervised linear feature selective projection (FSP) for feature extraction with low-rank embedding and dual Laplacian regularization, with the aim to exploit the intrinsic relationship among data and suppress the impact of noise. Specifically, a projection matrix with an l2,1-norm regularization is introduced to project original high dimensional data points into a new subspace with lower dimension, where the l2,1-norm regularization can endow the projection with good interpretability. We deploy a coefficient matrix with low rank constraint to reconstruct the data points and the l2,1-norm is imposed to regularize the data reconstruction errors in the low-dimensional subspace and make FSP robust to noise. Furthermore, a dual graph Laplacian regularization term is imposed on the low dimensional data and data reconstruction matrix for preserving the local manifold geometrical structure of data. Finally, an alternatively iterative algorithm is carefully designed for solving the proposed optimization model. Theoretical convergence and computational complexity analysis of the algorithm are also provided. Comprehensive experiments on various benchmark datasets have been carried out to evaluate the performance of the proposed FSP. As indicated, our algorithm significantly outperforms other state-of-the-art methods for feature extraction.

Incomplete multi-view clustering (IMVC) optimally fuses multiple pre-specified incomplete views to improve clustering performance. Among various excellent solutions, the recently proposed multiple kernel k-means with incomplete kernels (MKKM-IK) forms a benchmark, which redefines IMVC as a joint optimization problem where the clustering and kernel matrix imputation tasks are alternately performed until convergence. Though demonstrating promising performance in various applications, we observe that the manner of kernel matrix imputation in MKKM-IK would incur intensive computational and storage complexities, overcomplicated optimization and limitedly improved clustering performance. In this paper, we propose an Efficient and Effective Incomplete Multi-view Clustering (EE-IMVC) algorithm to address these issues. Instead of completing the incomplete kernel matrices, EE-IMVC proposes to impute each incomplete base matrix generated by incomplete views with a learned consensus clustering matrix. We carefully develop a three-step iterative algorithm to solve the resultant optimization problem with linear computational complexity and theoretically prove its convergence. Further, we conduct comprehensive experiments to study the proposed EE-IMVC in terms of clustering accuracy, running time, evolution of the learned consensus clustering matrix and the convergence. As indicated, our algorithm significantly and consistently outperforms some state-of-the-art algorithms with much less running time and memory.

Multi-view unsupervised feature selection (MV-UFS) aims to select a feature subset from multi-view data without using the labels of samples. However, we observe that existing MV-UFS algorithms do not well consider the local structure of cross views and the diversity of different views, which could adversely affect the performance of subsequent learning tasks. In this paper, we propose a cross-view local structure preserved diversity and consensus semantic learning model for MV-UFS, termed CRV-DCL briefly, to address these issues. Specifically, we project each view of data into a common semantic label space which is composed of a consensus part and a diversity part, with the aim to capture both the common information and distinguishing knowledge across different views. Further, an inter-view similarity graph between each pairwise view and an intra-view similarity graph of each view are respectively constructed to preserve the local structure of data in different views and different samples in the same view. An l2,1-norm constraint is imposed on the feature projection matrix to select discriminative features. We carefully design an efficient algorithm with convergence guarantee to solve the resultant optimization problem. Extensive experimental study is conducted on six publicly real multi-view datasets and the experimental results well demonstrate the effectiveness of CRV-DCL.

Defocus blur detection aims to detect out-of-focus regions from an image. Although attracting more and more attention due to its widespread applications, defocus blur detection still confronts several challenges such as the interference of background clutter, sensitivity to scales and missing boundary details of defocus blur regions. To deal with these issues, we propose a deep neural network which recurrently fuses and refines multi-scale deep features (DeFusionNet) for defocus blur detection. We firstly utilize a fully convolutional network to extract multi-scale deep features. The features from bottom layers are able to capture rich low-level features for details preservation, while the features from top layers can characterize the semantic information to locate blur regions. These features from different layers are fused as shallow features and semantic features, respectively. After that, the fused shallow features are propagated to top layers for refining the fine details of detected defocus blur regions, and the fused semantic features are propagated to bottom layers to assist in better locating the defocus regions. The feature fusing and refining are carried out in a recurrent manner. Also, we finally fuse the output of each layer at the last recurrent step to obtain the final defocus blur map by considering the sensitivity to scales of the defocus degree. Experiments on two commonly used defocus blur detection benchmark datasets are conducted to demonstrate the superority of DeFusionNet when compared with other 10 competitors. Code and more results can be found at: http://tangchang.net.

This paper proposes a novel deep saliency detection network by recurrently aggregating and refining features in a cross-level and spatial attention-aware manner. In this way, the features integrated from multiple layers can be used to refine layer-wise features step by step and the complementary information in different layers can be fully captured for detecting salient objects with different scales, i.e., the features integrated from low-level layers can serve to refine the details of detected salient objects while the features integrated from high-level layers with semantic information can benefit the locating of salient objects. In addition, by considering that only partial regions of an image are salient, we embed a spatial attention-aware module to suppress the non-salient regions and highlight salient objects. Finally, different saliency detection results from different layers are fused to generate the final saliency map. Experimental results on five benchmark datasets demonstrate that our proposed method outperforms other 14 state-of-the-art competitors.

Multi-view clustering (MVC) optimally integrates complementary information from different views to improve clustering performance. Although demonstrating promising performance in many applications, we observe that most of existing methods directly combine multiple views to learn an optimal similarity for clustering. These methods would cause intensive computational complexity and over-complicated optimization. In this paper, we theoretically uncover the connection between existing k-means clustering and the alignment between base partitions and consensus partition. Based on this observation, we propose a simple but effective multi-view algorithm termed {Multi-view Clustering via Late Fusion Alignment Maximization (MVC-LFA)}. In specific, MVC-LFA proposes to maximally align the consensus partition with the weighted base partitions. Such a criterion is beneficial to significantly reduce the computational complexity and simplify the optimization procedure. Furthermore, we design a three-step iterative algorithm to solve the new resultant optimization problem with theoretically guaranteed convergence. Extensive experiments on five multi-view benchmark datasets demonstrate the effectiveness and efficiency of the proposed MVC-LFA.

Incomplete multi-view clustering optimally integrates a group of pre-specified incomplete views to improve clustering performance. Among various excellent solutions, multiple kernel $k$k-means with incomplete kernels forms a benchmark, which redefines the incomplete multi-view clustering as a joint optimization problem where the imputation and clustering are alternatively performed until convergence. However, the comparatively intensive computational and storage complexities preclude it from practical applications. To address these issues, we propose Late Fusion Incomplete Multi-view Clustering (LF-IMVC) which effectively and efficiently integrates the incomplete clustering matrices generated by incomplete views. Specifically, our algorithm jointly learns a consensus clustering matrix, imputes each incomplete base matrix, and optimizes the corresponding permutation matrices. We develop a three-step iterative algorithm to solve the resultant optimization problem with linear computational complexity and theoretically prove its convergence. Further, we conduct comprehensive experiments to study the proposed LF-IMVC in terms of clustering accuracy, running time, advantages of late fusion multi-view clustering, evolution of the learned consensus clustering matrix, parameter sensitivity and convergence. As indicated, our algorithm significantly and consistently outperforms some state-of-the-art algorithms with much less running time and memory.

Multilabel image annotation attracts a lot of research interest due to its practicability in multimedia and computer vision fields, while the need for a large amount of labeled training data to achieve promising performance makes it a challenging task. Fortunately, unlabeled and relevant data are widely available and these data can be used to serve the annotation task. To this end, we propose a novel adaptive hypergraph learning (AHL) method for multilabel image annotation in a semisupervised way, in which both the limited labeled data and abundant unlabeled data are utilized to facilitate the annotation performance. In detail, we seek a multilabel propagation scheme by learning a hypergraph which is used to preserve the local geometric structures of data in a high-order manner. Meanwhile, a feature projection is integrated into AHL to obtain a latent feature space where unlabeled instances can be effectively and robustly assigned with multiple labels. Experiments on six widely used image datasets are conducted to evaluate our model and the results demonstrate that the proposed AHL outperforms other state-of-the-art semisupervised methods.

With the ability to exploit the internal structure of data, graph-based models have received a lot of attention and have achieved great success in multiview subspace clustering for multimedia data. Most of the existing methods individually construct an affinity graph for each single view and fuse the result obtained from each single graph. However, the common representation shared by different views and the complementary diversity across these views are not efficiently exploited. In addition, noise and outliers are often mixed in original data, which adversely degenerate the clustering performance of many existing methods. In this paper, we propose addressing these issues by learning a joint affinity graph for multiview subspace clustering based on a low-rank representation with diversity regularization and a rank constraint. Specifically, a low-rank representation model is employed to learn a shared sample representation coefficient matrix to generate the affinity graph. At the same time, we use diversity regularization to learn the optimal weights for each view, which can suppress the redundancy and enhance the diversity among different feature views. In addition, the cluster number is used to promote affinity graph learning by using a rank constraint. The final clustering result is obtained by using normalized cuts on the learned affinity graph. An efficient algorithm based on an augmented Lagrangian multiplier with alternating direction minimization is carefully designed to solve the resulting optimization problem. Extensive experiments on various real-world datasets are conducted, and the results demonstrate well the effectiveness of the proposed algorithm.

Recent research indicates the critical importance of preserving local geometric structure of data in unsupervised feature selection (UFS), and the well studied graph Laplacian is usually deployed to capture this property. By using a squared l 2 -norm, we observe that conventional graph Laplacian is sensitive to noisy data, leading to unsatisfying data processing performance. To address this issue, we propose a unified UFS framework via feature self-representation and robust graph regularization , with the aim at reducing the sensitivity to outliers from the following two aspects: i) an l 2, 1 -norm is used to characterize the feature representation residual matrix; and ii) an l 1 -norm based graph Laplacian regularization term is adopted to preserve the local geometric structure of data. By this way, the proposed framework is able to reduce the effect of noisy data on feature selection. Furthermore, the proposed l 1 -norm based graph Laplacian is readily extendible, which can be easily integrated into other UFS methods and machine learning tasks with local geometrical structure of data being preserved. As demonstrated on ten challenging benchmark data sets, our algorithm significantly and consistently outperforms state-of-the-art UFS methods in the literature, suggesting the effectiveness of the proposed UFS framework.

This paper proposes three simple, compact yet effective representations of depth sequences, referred to respectively as dynamic depth images (DDI), dynamic depth normal images (DDNI), and dynamic depth motion normal images (DDMNI), for both isolated and continuous action recognition. These dynamic images are constructed from a segmented sequence of depth maps using hierarchical bidirectional rank pooling to effectively capture the spatial-temporal information. Specifically, DDI exploits the dynamics of postures over time, and DDNI and DDMNI exploit the 3-D structural information captured by depth maps. Upon the proposed representations, a convolutional neural network (ConvNet)-based method is developed for action recognition. The image-based representations enable us to fine-tune the existing ConvNet models trained on image data without training a large number of parameters from scratch. The proposed method achieved the state-of-art results on three large datasets, namely, the large-scale continuous gesture recognition dataset (means the Jaccard index 0.4109), the large-scale isolated gesture recognition dataset (59.21%), and the NTU RGB+D dataset (87.08% cross-subject and 84.22% cross-view) even though only the depth modality was used.

Scene flow describes the motion of 3D objects in real world and potentially could be the basis of a good feature for 3D action recognition. However, its use for action recognition, especially in the context of convolutional neural networks (ConvNets), has not been previously studied. In this paper, we propose the extraction and use of scene flow for action recognition from RGB-D data. Previous works have considered the depth and RGB modalities as separate channels and extract features for later fusion. We take a different approach and consider the modalities as one entity, thus allowing feature extraction for action recognition at the beginning. Two key questions about the use of scene flow for action recognition are addressed: how to organize the scene flow vectors and how to represent the long term dynamics of videos based on scene flow. In order to calculate the scene flow correctly on the available datasets, we propose an effective self-calibration method to align the RGB and depth data spatially without knowledge of the camera parameters. Based on the scene flow vectors, we propose a new representation, namely, Scene Flow to Action Map (SFAM), that describes several long term spatio-temporal dynamics for action recognition. We adopt a channel transform kernel to transform the scene flow vectors to an optimal color space analogous to RGB. This transformation takes better advantage of the trained ConvNets models over ImageNet. Experimental results indicate that this new representation can surpass the performance of state-of-the-art methods on two large public datasets.

Image-based salient object detection is a useful and important technique, which can promote the efficiency of several applications such as object detection, image classification/retrieval, object co-segmentation, and content-based image editing. In this letter, we present a novel weighted low-rank matrix recovery (WLRR) model for salient object detection. In order to facilitate efficient salient objects-background separation, a high-level background prior map is estimated by employing the property of the color, location, and boundary connectivity, and then this prior map is ensembled into a weighting matrix which indicates the likelihood that each image region belongs to the background. The final salient object detection task is formulated as the WLRR model with the weighting matrix. Both quantitative and qualitative experimental results on three challenging datasets show competitive results as compared with 24 state-of-the-art methods.

This paper proposes three simple, compact yet effective representations of depth sequences, referred to respectively as Dynamic Depth Images (DDI), Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images (DDMNI). These dynamic images are constructed from a sequence of depth maps using bidirectional rank pooling to effectively capture the spatial-temporal information. Such image-based representations enable us to fine-tune the existing ConvNets models trained on image data for classification of depth sequences, without introducing large parameters to learn. Upon the proposed representations, a convolutional Neural networks (ConvNets) based method is developed for gesture recognition and evaluated on the Large-scale Isolated Gesture Recognition at the ChaLearn Looking at People (LAP) challenge 2016. The method achieved 55.57% classification accuracy and ranked 2ndplace in this challenge but was very close to the best performance even though we only used depth data.

Blur exists in many digital images, it can be mainly categorized into two classes: defocus blur which is caused by optical imaging systems and motion blur which is caused by the relative motion between camera and scene objects. In this letter, we propose a simple yet effective automatic blurred image region detection method. Based on the observation that blur attenuates high-frequency components of an image, we present a blur metric based on the log averaged spectrum residual to get a coarse blur map. Then, a novel iterative updating mechanism is proposed to refine the blur map from coarse to fine by exploiting the intrinsic relevance of similar neighbor image regions. The proposed iterative updating mechanism can partially resolve the problem of differentiating an in-focus smooth region and a blurred smooth region. In addition, our iterative updating mechanism can be integrated into other image blurred region detection algorithms to refine the final results. Both quantitative and qualitative experimental results demonstrate that our proposed method is more reliable and efficient compared to various state-of-the-art methods.

This paper proposes a new method, i.e., weighted hierarchical depth motion maps (WHDMM) + three-channel deep convolutional neural networks (3ConvNets), for human action recognition from depth maps on small training datasets. Three strategies are developed to leverage the capability of ConvNets in mining discriminative features for recognition. First, different viewpoints are mimicked by rotating the 3-D points of the captured depth maps. This not only synthesizes more data, but also makes the trained ConvNets view-tolerant. Second, WHDMMs at several temporal scales are constructed to encode the spatiotemporal motion patterns of actions into 2-D spatial structures. The 2-D spatial structures are further enhanced for recognition by converting the WHDMMs into pseudocolor images. Finally, the three ConvNets are initialized with the models obtained from ImageNet and fine-tuned independently on the color-coded WHDMMs constructed in three orthogonal planes. The proposed algorithm was evaluated on the MSRAction3D, MSRAction3DExt, UTKinect-Action, and MSRDailyActivity3D datasets using cross-subject protocols. In addition, the method was evaluated on the large dataset constructed from the above datasets. The proposed method achieved 2-9% better results on most of the individual datasets. Furthermore, the proposed method maintained its performance on the large dataset, whereas the performance of existing methods decreased with the increased number of actions.

Covariance matrix has recently received increasing attention in computer vision by leveraging Riemannian geometry of symmetric positive-definite (SPD) matrices. Originally proposed as a region descriptor, it has now been used as a generic representation in various recognition tasks. However, covariance matrix has shortcomings such as being prone to be singular, limited capability in modeling complicated feature relationship, and having a fixed form of representation. This paper argues that more appropriate SPD-matrix-based representations shall be explored to achieve better recognition. It proposes an open framework to use the kernel matrix over feature dimensions as a generic representation and discusses its properties and advantages. The proposed framework significantly elevates covariance representation to the unlimited opportunities provided by this new representation. Experimental study shows that this representation consistently outperforms its covariance counterpart on various visual recognition tasks. In particular, it achieves significant improvement on skeleton-based human action recognition, demonstrating the state-of-the-art performance over both the covariance and the existing non-covariance representations.

In this paper, we propose to adopt ConvNets to recognize human actions from depth maps on relatively small datasets based on Depth Motion Maps (DMMs). In particular, three strategies are developed to effectively leverage the capability of ConvNets in mining discriminative features for recognition. Firstly, different viewpoints are mimicked by rotating virtual cameras around subject represented by the 3D points of the captured depth maps. This not only synthesizes more data from the captured ones, but also makes the trained ConvNets view-tolerant. Secondly, DMMs are constructed and further enhanced for recognition by encoding them into Pseudo-RGB images, turning the spatial-temporal motion patterns into textures and edges. Lastly, through transferring learning the models originally trained over ImageNet for image classification, the three ConvNets are trained independently on the color-coded DMMs constructed in three orthogonal planes. The proposed algorithm was extensively evaluated on MSRAction3D, MSRAction3DExt and UTKinect-Action datasets and achieved the state-of-the-art results on these datasets.