Multi-view clustering (MVC) aims to uncover the latent structure of multi-view data by learning view-common and view-specific information. Although recent studies have explored hyperbolic representations for better tackling the representation gap between different views, they focus primarily on instance-level alignment and neglect global semantic consistency, rendering them vulnerable to view-specific information (e.g., noise and cross-view discrepancies). To this end, this paper proposes a novel Wasserstein-Aligned Hyperbolic (WAH) framework for multi-view clustering. Specifically, our method exploits a view-specific hyperbolic encoder for each view to embed features into the Lorentz manifold for hierarchical semantic modeling. Whereafter, a global semantic loss based on the hyperbolic sliced-Wasserstein distance is introduced to align manifold distributions across views. This is followed by soft cluster assignments to encourage cross-view semantic consistency. Extensive experiments on multiple benchmarking datasets show that our method can achieve SOTA clustering performance.

Visual object tracking has witnessed continuous advances in recent years along with the exciting developments in backbone networks. In general, all advanced solutions adhere to the template-based tracking framework, which exhibits powerful representative capacity gained via offline training. However, when the target undergoes appearance changes or occlusion, the tracker, which relies on a fixed template defined in the initial frame, struggles to locate it accurately in such complex situations. To achieve online adaptation, recent studies have introduced dynamic templates. Typically, the adopted solution is to compute reliability scores in the traditional Euclidean space to assess the confidence of the dynamic template. However, the Euclidean metric is unreliable to some extent in high-dimensional feature spaces, potentially resulting in a negative impact by involving incorrect dynamic templates. To overcome this problem, we exploit the compact geometric representation capacity of the Symmetric Positive Definite (SPD) manifold to design a novel score prediction module for the tracker update (SPD-Updater). By switching to an SPD manifold metric, we obtain a more accurate and stable dynamic template, thereby enhancing the model capacity to handle complex situations. To validate the reliability of manifold metric in tracking models, we conduct experiments with trackers using different backbones. The experimental results on LaSOT, GOT-10k, TrackingNet, and UAV123 demonstrate the effectiveness of our approach, reflecting the merit of the SPD metric in online tracking adaptation.

Evaluation is essential in image fusion research, yet most existing metrics are directly borrowed from other vision tasks without proper adaptation. These traditional metrics, often based on complex image transformations, not only fail to capture the true quality of the fusion results but also are computationally demanding. To address these issues, we propose a unified evaluation framework specifically tailored for image fusion. At its core is a lightweight network designed efficiently to approximate widely used metrics, following a divide-and-conquer strategy. Unlike conventional approaches that directly assess similarity between fused and source images, we first decompose the fusion result into infrared and visible components. The evaluation model is then used to measure the degree of information preservation in these separated components, effectively disentangling the fusion evaluation process. During training, we incorporate a contrastive learning strategy and inform our evaluation model by perceptual scene assessment provided by a large language model. Last, we propose the first consistency evaluation framework, which measures the alignment between image fusion metrics and human visual perception, using both independent no-reference scores and downstream tasks performance as objective references. Extensive experiments show that our learning-based evaluation paradigm delivers both superior efficiency (up to 1,000 times faster) and greater consistency across a range of standard image fusion benchmarks.

In most existing multi-view modeling scenarios, cross-view correspondence (CVC) between instances of the same target from different views, like paired image-text data, is a crucial prerequisite for effortlessly deriving a consistent representation. Nevertheless, this premise is frequently compromised in certain applications, where each view is organized and transmitted independently, resulting in the view-unaligned problem (VuP). Restoring CVC of unaligned multi-view data is a challenging and highly demanding task that has received limited attention from the research community. To tackle this practical challenge, we propose to integrate the permutation derivation procedure into the bipartite graph paradigm for view-unaligned clustering, termed Probabilistically Aligned View-unaligned Clustering with Adaptive Template Selection (PAVuC-ATS). Specifically, we learn consistent anchors and view-specific graphs by the bipartite graph, and derive permutations applied to the unaligned graphs by reformulating the alignment between two latent representations as a 2-step transition of a Markov chain with adaptive template selection, thereby achieving the probabilistic alignment. The convergence of the resultant optimization problem is validated both experimentally and theoretically. Extensive experiments on six benchmark datasets demonstrate the superiority of the proposed PAVuC-ATS over the baseline methods.

To address the challenge of autonomous navigation for mobile robots in complex environments, many recent path planning methods employ a two-layer motion framework that integrates both global and local planning. However, global planning algorithms based on evolutionary algorithms often yield suboptimal paths, primarily due to premature convergence and wasted iterations in infeasible regions. Meanwhile, local planning algorithms, particularly the dynamic window approach (DWA) and its variants, are frequently tailored to specific scenarios, limiting their generalizability. To address these issues, this paper proposes an effective path planning method, namely M2PP. The global planning algorithm in M2PP enhances particle swarm optimization (PSO) by introducing multiple new search phases, creating a multi-phase PSO that can quickly identify feasible global paths and thoroughly explore the sampling space. To effectively handle multiple scenarios in a more generalized manner, the local planning component, multi-scenario adaptative DWA, integrates two novel terms into its cost function, enhancing both static and dynamic obstacle avoidance. Extensive simulations demonstrate that multi-phase PSO exhibits strong performance and robustness, especially in highly complex scenarios, and multi-scenario adaptative DWA can generate safe and efficient local trajectories in hazardous conditions. Additionally, experiments deploying the M2PP method on a real-world mobile robot further confirm its feasibility.

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) is a challenging task that focuses on retrieving relevant images based on sketch queries at the instance level. Most existing works tend to make cross-modality interactions between features at the global level and treat each feature equally when calculating similarity. However, this equal treatment may reduce the contribution of fine-grained information. To address these challenges, we propose a novel Cross-Modality Neighborhood Attention Network, named CMNA-Net. Specifically, we develop a novel CMNA mechanism in the proposed method to adjust the importance of modality-agnostic features, which not only performs neighborhood attention operation on the current modality but also performs cross-neighborhood attention on both modalities. Additionally, the edge map is innovatively introduced to reduce the modality gap. Furthermore, this work proposes a Neighborhood Weighted module that integrates the local feature weight scheme and top-k feature selection strategy, yielding more discriminative features tailored to fine-grained retrieval. Extensive experiments on three benchmark datasets demonstrate that our proposed CMNA-Net can achieve better performance compared to some existing state-of-the-art methods in FG-SBIR scenarios. The code of our method will be available at https://github.com/li1changxing/CMNA-Net.

Online visual object tracking fundamentally constitutes a continual learning challenge, demanding persistent adaptation to target variations within dynamic video streams, while preserving critical features to prevent forgetting. Existing template-based trackers are particularly vulnerable to severe target deformation and drastic background changes in real-world scenarios. Recent explorations enhance adaptability through local update strategies—such as dynamic templates, appearance tokens, and parameter fine-tuning—to address rapid appearance variations. However, these methods inherently propagate target appearance changes without explicit modelling and prioritise short-term adaptation over long-term global representation. Consequently, they fail to balance initial target features with current observations, leading to tracking failure in long-term scenarios. To overcome these limitations, we propose DG-Track, an adaptive continual learning framework leveraging Grassmannian manifold geometry. Specifically, we represent target appearance within a Grassmannian affine subspace and perform continual adaptation via incremental learning. Compared to Euclidean geometry, the Grassmannian manifold captures nonlinear appearance representations, yielding more compact and geometrically consistent temporal dynamics modelling. Furthermore, an adaptive forgetting module dynamically regulates the interplay between the current observed subspace and the initial template, ensuring stable long-term tracking. DG-Track is a plug-and-play solution for online tracking, adding no learnable parameters. Comprehensive experiments with diverse baseline trackers on LaSOT, GOT-10k, TrackingNet, and UAV123 validate the efficacy of our continual learning framework and Grassmannian manifold geometry in enhancing visual object tracking performance. Code is available at https://github.com/xiaoqing0825/DGTrack.

The increasing frequency of global wildfires has led to the destruction of vast forests and wetlands. Non-contact remote sensing technologies provide an effective means for accurate burned area segmentation (BAS). However, existing BAS methods often treat each image independently, focusing primarily on local pixel contexts while neglecting the broader semantic consistency of burned regions across different scenes. The lack of global context modeling limits their robustness, as burned areas typically exhibit distinctive and consistent visual characteristics such as color and texture across diverse environments. To address this limitation, we propose a Self-image and Cross-image Consistency Learning (SCCL) framework, which captures both local pixel-level relationships within a single image and global semantic dependencies across multiple images. By enforcing consistent and compact representations of burned regions within and across images, SCCL enhances segmentation robustness under varying weather and terrain conditions. Additionally, to refine boundary delineation between burned and unburned areas, we introduce a Burned Edge Injector (BEI) and an Edge-Injected Decoder (EID). We further construct two large-scale BAS benchmark datasets, BAS-AUS and BAS-EUR, for comprehensive evaluation. Experiments on these benchmarks demonstrate that our method achieves state-of-the-art performance, significantly outperforming previous approaches, with MAE reduced to 0.017 and 0.016, respectively. The new BAS benchmarks and code are available at https://github.com/VisionVerse/SCCL.

This paper studies the problem of text-video retrieval, where the goal is to learn accurate cross-modal alignment between videos and text. This problem is challenging because of the matching ambiguity caused by the inherent gap between the heterogeneous video and text modalities. In particular, the differences in the information granularity and abstraction levels between the two modalities hinder a reliable sample-level alignment. Moreover, redundant visual content, sparse textual descriptions, and temporal variability in videos introduce additional uncertainty, resulting in ambiguous matching and suboptimal performance. In this paper, we propose a novel method named Probabilistic Embeddings with Evidence Learning and Refinement (PE2LR), which models video-text pairs as probability distributions and captures uncertainty through the evidence theory. Specifically, we perform distribution-level representation learning to resolve the semantic ambiguity of video-text pairs. To improve the alignment further, we introduce a distribution-based embedding refinement module to ameliorate the semantic consistency across modalities. The proposed PE2LR is able to pull positive sample pairs closer in the embedding space, while pushing the negative pairs apart. Comprehensive experiments on several benchmark datasets (including MSRVTT, DiDeMo, and ActivityNet Captions) demonstrate that our PE2LR achieves state-of-the-art search performance.

Diffusion Probabilistic Models (DPMs) have recently demonstrated considerable potential for single image super-resolution (SISR) by utilizing a conditional generation process that transforms Gaussian noise into high-resolution (HR) images based on low-resolution (LR) inputs. Current Image-Conditional DPMs (icDPMs) have demonstrated promising results by leveraging LR images as a condition to guide the generation of HR images. However, icDPMs fail to effectively integrate LR images and other conditional information to generate accurate and natural output. To address this issue, we propose an Integrated Conditional Diffusion Model for Single Image Super-Resolution (ICDSR). Our approach encodes the LR image as a condition to generate the prior feature, simultaneously integrating it with timestep information to establish intermediate constraints. To further enhance these constraints, we designed a multi-scale guidance structure for the U-shaped concatenation of the diffusion model during the integration of conditions. This constraint serves as multi-scale guidance specifically designed for the U-shaped concatenation of the diffusion model during the integration of conditions. Specifically, multi-scale integrated information is injected into the diffusion model basic block, informing about the coarse structure of the sharp image at the intermediate layers with spatially adaptive conditions. Additionally, ICDSR employs a lightweight U-Net to provide initial guidance and leverages the diffusion model to learn residual guidance for faster convergence. Extensive experiments on facial and general benchmarks, including the CelebA and DIV2K datasets, demonstrate that ICDSR surpasses existing methods, achieving state-of-the-art perceptual quality while maintaining competitive distortion metrics.

In recent years, CLIP-based text-video retrieval has made significant progress. Most existing text-video retrieval methods usually directly align global video-level and sentence-level features without establishing intrinsic connections between different modalities. Furthermore, due to the asymmetry between visual and textual information, the symmetrical modeling also has inherent limitations that may affect the search performance. To mitigate these problems, we propose an asymmetric hierarchical text-video retrieval framework based on clustering prototypes. Specifically, beyond the commonly used global features, we generate video instance prototypes using clustering algorithms and derive video object prototypes from video patch tokens. The text instance prototypes can be obtained by the developed Text Prototype Generation module (TPG). Besides, the Joint-Interaction Module (JIM) and the coarse-grained instance alignment scheme are proposed to coordinate different modalities. Furthermore, the cross-correlation matrix among instances is proposed, which can further reduce the heterogeneity between video and text modalities. Extensive experiments on several datasets (including MSRVTT, and MSVD) demonstrate that our method outperforms some existing state-of-the-art methods. The source code is available at:https://github.com/rzheng77/PAHM-text-video-retrieval.

Text-video retrieval has advanced significantly by thanks to the merits of large-scale contrastive languageimage pre-training. Recently, prompt tuning has emerged as a parameter-efficient strategy to solve the substantial training costs associated with these models. However, existing studies are generally confined to simple tuning strategies, without fully exploring the potential of prompts. Unlike its application to singlemodal tasks, prompt tuning in cross-modal learning involves not only the adaptation to specific tasks but also the challenge of dealing with the intricate interaction of semantic information. These aspects remain insufficiently explored, thereby limiting retrieval performance. To this end, we introduce a comprehensive framework for harnessing the full potential of prompts in textvideo retrieval. Specifically, we explore prompts in three respects: i) Task adaptation: we design appropriate prompts to guide the fine-tuning of pre-trained models to perform text-video retrieval tasks. ii) Temporal modelling: to enable temporal modelling, while protecting the original knowledge from being corrupted, we propose a prompt rolling module that enables effective temporal processing without adding extra training parameters. iii) The sampling scope prediction: We introduce a query expansion strategy that exploits the rich semantic information conveyed by multimodal prompt features to define the sampling scope more precisely. Extensive experiments on four mainstream datasets, i.e., MSR-VTT, ActivityNet, DiDeMo, and LSMDC, demonstrate that our method achieves state-of-the-art performance.

In multi-modal imaging scenarios, the misalignment of images presents a persistent challenge. Conventional image fusion algorithms, aiming to enhance the performance of downstream vision tasks, presuppose strictly registered inputs to achieve satisfactory results. To relax this assumption, a common approach is to register the images first; however, existing multi-modal registration methods are often hindered by complex architectures and a heavy reliance on semantic information. This article proposes BusRef, a unified framework that jointly addresses image registration and fusion, with a specific focus on the Infrared-Visible Image Registration and Fusion (IVRF) task. Within this framework, unaligned image pairs are processed through three sequential stages: coarse registration, fine registration, and fusion. We demonstrate that this integrated approach enables more robust and accurate IVRF. Key to our framework is a novel training and evaluation strategy that employs masks to mitigate the influence of non-reconstructible regions on the loss function, thereby significantly improving the model’s accuracy and robustness. Furthermore, we introduce a gradient-aware fusion network designed to effectively preserve complementary information from both modalities. Comprehensive experiments demonstrate that BusRef achieves superior performance when compared against various state-of-the-art registration and fusion algorithms. Our code is available at https://github.com/Yukarizz/BusReF .

With the explosive growth of video data, video moment retrieval (VMR) has attracted increasing attention due to its ability to localize semantically relevant moments in untrimmed videos. However, existing VMR approaches usually rely on annotated video-text correspondences or temporal annotations, both of which require significant human effort and are costly to scale. Even worse, the inherent subjectivity in manual labeling often introduces inconsistencies into the training data, further complicating the issue. In this article, we investigate the problem of Zero-Shot Video Moment Retrieval (ZS-VMR) and develop a novel method, Resilient Semantic Pseudo-Text Modeling (RSPT). The core of RSPT is to construct semantically rich pseudo-text embeddings through visually guided perturbations. Specifically, RSPT first generates initial pseudo-texts by injecting random noise into visual features and then learns adaptive noise weights by modeling the correlations between these pseudo-texts and visual features. This enables the generation of diverse and semantically aligned representations from multiple perspectives. To ensure alignment with visual semantics and suppress irrelevant noise, RSPT introduces a quality-aware contrastive loss that regularizes the semantic boundaries of pseudo-texts. Extensive experiments on Charades-STA and ActivityNet-Captions show that RSPT outperforms existing competitive baselines, validating its efficacy. Code is available at https://github.com/dmcsy/RSPT .

Drug Target Interaction (DTI) prediction has witnessed promising performance boosts accompanied by advanced multimodal feature extraction. However, existing approaches suffer from two main difficulties. First, the complex protein structures cannot be well represented by current protein-sequence-based feature extractors. Second, the gap between protein and drug features increases the vulnerability of the obtained classifier thus degrading the prediction robustness. To address these issues, we propose a novel R-DTI method by exploring the second-order relevance in both protein structural feature extraction and DTI prediction phases. Specifically, we construct a pre-trained structural feature extractor that mines the atomic relevance of each amino acid. Then, an inter-feature structure-preserved Riemannian network is designed to expand the existing protein extraction patterns. To improve the prediction robustness, we also develop a Riemannian classifier that uses the second-order protein-drug relevance with a unified feature space. Extensive experimental results demonstrate the merits and superiority of our R-DTI against the state-of-the-art, achieving 1.4% and 1.9% higher AUC-ROC on the BindingDB and DrugBank datasets, respectively.

This paper presents the outcomes of the PETS2025 challenge, held in conjunction with AVSS 2025 and sponsored by the EU-funded EURMARS project. The challenge introduces a novel maritime surveillance dataset comprising image sequences captured by diverse multi-altitude, multimodal sensors, reflecting the real-world multi-authority environment. The key tasks include: (1) object detection using various sensors across different platforms (ground-based and low-altitude aerial) and spectral ranges (visible, thermal, ultraviolet (UV), and short-wave infrared (SWIR)); (2) long-term tracking of targets in maritime environments spanning both sea and land; and (3) approximating target geolocations by using sensor imagery and telemetry data. Performance evaluations of results submitted by 12 international participants are discussed. The results show the effectiveness of these submissions and highlight ongoing challenges posed by heterogeneous sensors and complex environments. These challenges emphasise the need to further improve detection, tracking, and geolocation approximation for maritime and coastal surveillance.

Advanced image fusion methods mostly prioritise high-level missions, where task interaction struggles with semantic gaps, requiring complex bridging mechanisms. In contrast, we propose to leverage low-level vision tasks from digital photography fusion, allowing for effective feature interaction through pixel-level supervision. This new paradigm provides strong guidance for unsupervised multimodal fusion without relying on abstract semantics, enhancing task-shared feature learning for broader applicability. Owning to the hybrid image features and enhanced universal representations, the proposed GIFNet supports diverse fusion tasks, achieving high performance across both seen and unseen scenarios with a single model. Uniquely, experimental results reveal that our framework also supports single-modality enhancement, offering superior flexibility for practical applications. Our code will be available at https://github.com/AWCXV/GIFNet.

Covariance matrices have proven highly effective across many scientific fields. Since these matrices lie within the Symmetric Positive Definite (SPD) manifold—a Riemannian space with intrinsic non-Euclidean geometry, the primary challenge in representation learning is to respect this underlying geometric structure. Drawing inspiration from the success of Euclidean deep learning, researchers have developed neural networks on the SPD manifolds for more faithful covariance embedding learning. A notable advancement in this area is the implementation of Riemannian batch normalization (RBN), which has been shown to improve the performance of SPD network models. Nonetheless, the Riemannian metric beneath the existing RBN might fail to effectively deal with the ill-conditioned SPD matrices (ICSM), undermining the effectiveness of RBN. In contrast, the Bures-Wasserstein metric (BWM) demonstrates superior performance for ill-conditioning. In addition, the recently introduced Generalized BWM (GBWM) parameterizes the vanilla BWM via an SPD matrix, allowing for a more nuanced representation of vibrant geometries of the SPD manifold. Therefore, we propose a novel RBN algorithm based on the GBW geometry, incorporating a learnable metric parameter. Moreover, the deformation of GBWM by matrix power is also introduced to further enhance the representational capacity of GBWMbased RBN. Experimental results on different datasets validate the effectiveness of our proposed method. The code is available at https://github.com/jjscc/GBWBN.

With the rapid advancement of artificial intelligence, the field of Artificial Intelligence Generated Content (AIGC) has seen significant growth. As AI-generated images (AIGIs) become increasingly prevalent, the AIGC image quality assessment(AIGCIQA) has gained critical importance. However, traditional image quality assessment methods struggle to account for the complex relationship between generated images and their corresponding text prompts, leading to suboptimal performance in AIGCIQA tasks. Additionally, the scarcity of subjective annotation data within AIGCIQA datasets limits the effectiveness of deep learning models. To address these challenges, we propose a novel self-supervised task that utilizes intermediate image sequences generated during Text-to-Image (T2I) model generation as the pre-training data. We use the number of model iterations as pseudo-labels based on the positive correlation between image quality and the number of model iterations. Due to the relative coarseness of this pseudo-label as a supervised signal, we also introduce a linear interpolation method for optimization. Additionally, we designed a framework that effectively fuses image and text features. The framework considers the dynamic fading properties of intermediate image sequences similar to video clips. It utilizes a specialized text prompt template and extracts image features through a cross-attention mechanism, which significantly improves model performance. Experimental results show that the proposed method achieves state-of-the-art performance on three mainstream AIGCIQA datasets.

The shared topology of human skeletons motivated the recent investigation of graph convolutional network (GCN) solutions for action recognition. However, most of the existing GCNs rely on the binary connection of two neighboring vertices (joints) formed by an edge (bone), overlooking the potential of constructing multi-vertex convolution structures. Although some studies have attempted to utilize hyper-graphs to represent the topology, they rely on a fixed construction strategy, which limits their adaptivity in uncovering the intricate latent relationships within the action. In this paper, we address this oversight and explore the merits of an adaptive hyper-graph convolutional network (Hyper-GCN) to achieve the aggregation of rich semantic information conveyed by skeleton vertices. In particular, our Hyper-GCN adaptively optimises the hyper-graphs during training, revealing the action-driven multi-vertex relations. Besides, virtual connections are often designed to support efficient feature aggregation, implicitly extending the spectrum of dependencies within the skeleton. By injecting virtual connections into hyper-graphs, the semantic clues of diverse action categories can be highlighted. The results of experiments conducted on the NTU-60, NTU-120, and NW-UCLA datasets demonstrate the merits of our Hyper-GCN, compared to the state-of-the-art methods. The code is available at https://github.com/6UOOON9/Hyper-GCN.

Global Covariance Pooling (GCP) has been demonstrated to improve the performance of Deep Neural Networks (DNNs) by exploiting second-order statistics of high-level representations. GCP typically performs classification of the covariance matrices by applying matrix function normalization, such as matrix logarithm or power, followed by a Euclidean classifier. However, covariance matrices inherently lie in a Riemannian manifold, known as the Symmetric Positive Definite (SPD) manifold. The current literature does not provide a satisfactory explanation of why Euclidean classifiers can be applied directly to Riemannian features after the normalization of the matrix power. To mitigate this gap, this paper provides a comprehensive and unified understanding of the matrix logarithm and power from a Riemannian geometry perspective. The underlying mechanism of matrix functions in GCP is interpreted from two perspectives: one based on tangent classifiers (Euclidean classifiers on the tangent space) and the other based on Riemannian classifiers. Via theoretical analysis and empirical validation through extensive experiments on fine-grained and large-scale visual classification datasets, we conclude that the working mechanism of the matrix functions should be attributed to the Riemannian classifiers they implicitly respect. The code is available at https://github.com/GitZH-Chen/RiemGCP.git.

Several Riemannian manifolds in machine learning, such as Symmetric Positive Definite (SPD), Grassmann, spherical, and hyperbolic manifolds, have been proven to admit gyro structures, thus enabling a principled and effective extension of Euclidean Deep Neural Networks (DNNs) to manifolds. Inspired by this, this study introduces a general Riemannian Batch Normalization (RBN) framework on gyrogroups, termed GyroBN. We identify the least requirements to guarantee GyroBN with theoretical control over sample statistics, referred to as \textit{pseudo-reduction} and \textit{gyroisometric gyrations}, which are satisfied by all the existing gyrogroups in machine learning. Besides, our GyroBN incorporates several existing normalization methods, including the one on general Lie groups and different types of RBN on the non-group SPD geometry. Lastly, we instantiate our GyroBN on the Grassmannian and hyperbolic spaces. Experiments on the Grassmannian and hyperbolic networks demonstrate the effectiveness of our GyroBN. The code is available at https://github.com/GitZH-Chen/GyroBN.git.

When shooting electronic screens, moiré patterns usually appear in captured images, which seriously affects the image quality. Existing image demoiréing methods face great challenges in removing large and heavy moiré. To address the issue, we propose to utilize Dual Camera fusion for Image Demoiréing (DCID), i.e., using the ultra-wide-angle (UW) image to assist the moiré removal of wide-angle (W) image. This is inspired by two motivations: (1) the two lenses are commonly equipped with modern smartphones, (2) the UW image generally can provide normal colors and textures when moiré exists in the W image mainly due to their different focal lengths. In particular, we propose an efficient DCID method, where a lightweight UW image encoder is integrated into an existing demoiréing network and a fast two-stage image alignment manner is present. Moreover, we construct a large-scale real-world dataset with diverse mobile phones and monitors, containing about 9,000 samples. Experiments on the dataset show our method performs better than state-of-the-art methods. Code and dataset are available at https://github.com/Mrduckk/DCID.

Batch Normalisation (BN) is widely used in conventional deep neural network training to harmonise the input-output distributions for each batch of data. However, federated learning, a distributed learning paradigm, faces the challenge of dealing with non-independent and identically distributed data among the client nodes. Due to the lack of a coherent methodology for updating BN statistical parameters, standard BN degrades the federated learning performance. To this end, it is urgent to explore an alternative normalisation solution for federated learning. In this work, we resolve the dilemma of the BN layer in federated learning by developing a customised normalisation approach, Hybrid Batch Normalisation (HBN). HBN separates the update of statistical parameters (*i.e.*, means and variances used for evaluation) from that of learnable parameters (*i.e.*, parameters that require gradient updates), obtaining unbiased estimates of global statistical parameters in distributed scenarios. In contrast with the existing solutions, we emphasise the supportive power of global statistics for federated learning. The HBN layer introduces a learnable hybrid distribution factor, allowing each computing node to adaptively mix the statistical parameters of the current batch with the global statistics. Our HBN can serve as a powerful plugin to advance federated learning performance. It reflects promising merits across a wide range of federated learning settings, especially for small batch sizes and heterogeneous data. Code is available at https://github.com/Hongyao-Chen/HybridBN.

Riemannian neural networks, which generalize the deep learning paradigm to non-Euclidean geometries, have garnered widespread attention across diverse applications in artificial intelligence. Among these, the representative attention models have been studied on various non-Euclidean spaces to geometrically capture the spatiotemporal dependencies inherent in time series data, e.g., electroencephalography (EEG). Recent studies have highlighted the full-rank correlation matrix as an advantageous alternative to the covariance matrix for data representation, owing to its invariance to the scale of variables. Motivated by these advancements, we propose the Correlation Attention Network (CorAtt) tailored for full-rank correlation matrices and implement it under the permutation-invariant and computationally efficient Off-Log and Log-Scaled geometries, respectively. Extensive evaluations on three benchmarking EEG datasets provide substantial evidence for the effectiveness of our introduced CorAtt. The code and supplementary material can be found at https://github.com/ChenHu-ML/CorAtt.

Symmetric Positive Definite (SPD) matric has been proven to be an effective feature descriptor in the realm of artificial intelligence, as it can encode spatiotemporal statistical information of data on a curved Riemannian manifold, i.e., SPD manifold. Although existing Riemannian neural networks have demonstrated superiority in many scientific fields, the inherent reliance on labels within supervised learning renders them susceptible to label errors. Besides, it is insufficient to depend solely on labels to learn effective feature distributions in some complicated data scenarios. Drawing inspiration from the considerable achievements of contrastive learning (CL) across diverse tasks, we extend the conventional CL paradigm to the context of SPD manifolds, which we denote SymCL, paving the way for a novel approach in SPD matrix-based visual classification. Furthermore, we inject a Riemannian triplet loss-based Riemannian metric learning (RML) into the designed SPD manifold CL framework for the sake of improving the discrimination of the learned geometric representations. Extensive experimental results on four datasets verify the effectiveness of the proposed algorithm.

Learning representations on the Grassmannian manifold is popular in quite a few visual classification tasks. With the development of deep learning techniques, several neural networks have recently emerged for processing subspace data. However, the diversely changed appearance of the signal data (video clips and image sets), makes it impossible for the existing Grassmannian networks (GrasNets) that rely on a single cross-entropy loss for end-to-end training to learn effective geometric representations, especially for complicated visual scenarios. To solve this problem, a Riemannian triplet loss-based Riemannian metric learning mechanism is introduced to the original GrasNet, which can explicitly encode and learn the characteristics of the intra- and inter-class data distributions conveyed by the input data during network training. Additionally, given the existence of intra-class diversity and inter-class ambiguity of the input data, we propose a hard sample reward strategy (HSR) to further improve the discriminability of the learned network embedding. Extensive experimental results obtained on four benchmarking datasets demonstrate the effectiveness of the proposed method.

Food plays a vital role in human health, and accurate nutrition estimation is crucial for guiding healthy dietary choices. Traditional biochemical-based assessment methods are often inefficient, costly, and impractical for daily use. With the continuous progress in computer vision, some vision-based nutrition estimation approaches have emerged, typically relying on RGB images alone or in combination with depth images to infer nutritional information. These methods have achieved promising performance and garnered considerable attention. However, these methods often ignore visually imperceptible ingredients such as oil, sugar, and salt, which may significantly influence the estimation of nutritional content. Besides, existing methods lack explicit mechanisms for modeling nutrient-specific information and guiding attention toward nutrition-relevant semantics. To solve the above two issues, we propose a novel ingredients-guided and nutrients-prompted nutrition estimation method. Our method adopts multi-scale feature fusion and integrates RGB and depth modalities to enhance visual representation learning. To account for invisible ingredients, we introduce an ingredients-guided strategy, which enhances the sensitivity to non-visible nutritional factors. Moreover, a nutrient-prompt mechanism is introduced to explicitly guide the focus of the model toward nutrient-relevant attributes during estimation. We validate our method on Nutrition5k, where it consistently outperforms existing state-of-the-art methods, demonstrating its efficacy.

Unifying multiple multi-modal visual object tracking (MMVOT) tasks draws increasing attention due to the complementary nature of different modalities in building robust tracking systems. Existing practices mix all data sensor types in a single training procedure, structuring a parallel paradigm from the data-centric perspective and aiming for a global optimum on the joint distribution of the involved tasks. However, the absence of a unified benchmark where all types of data coexist forces evaluations on separated benchmarks, causing inconsistency between training and testing, thus leading to performance degradation. To address these issues, this work advances in two aspects: A unified benchmark, coined as UniBench300, is introduced to bridge the inconsistency by incorporating multiple task data, reducing inference passes from three to one and cutting time consumption by 27%. The unification process is reformulated in a serial format, progressively integrating new tasks. In this way, the performance degradation can be specified as knowledge forgetting of previous tasks, which naturally aligns with the philosophy of continual learning (CL), motivating further exploration of injecting CL into the unification process. Extensive experiments conducted on two baselines and four benchmarks demonstrate the significance of UniBench300 and the superiority of CL in supporting a stable unification process. Moreover, while conducting dedicated analyses, the performance degradation is found to be negatively correlated with network capacity. Additionally, modality discrepancies contribute to varying degradation levels across tasks (RGBT > RGBD > RGBE in MMVOT), offering valuable insights for future multi-modal vision research. Source codes and the proposed benchmark is available at https://github.com/Zhangyong-Tang/UniBench300.

The Few-Shot Fine-Grained Multimodal Named Entity Recognition task (FewFMNER) aims to recognize the named entities and their fine-grained types in the text based on the social multimedia content of images and texts. With limited labeled data, FewFMNER can more accurately understand the semantics of multimodal content and support downstream applications in social networks. However, as the number of fine-grained types increases, two challenges will arise: (1) Entity boundary ambiguity and concept misjudgment. (2) Semantic confusion exists in fine-grained types. Therefore, we propose a novel Multi-agent Concept Check and Dual-path Reasoning (MCCDR) framework: (1) Entity boundary detection: Recognize named entities and filter concepts to obtain candidate named entity boundaries. (2) Entity type classification: Use dual-path reasoning and type definition to alleviate semantic confusion, and utilize a self-consistency strategy to fuse (entity, fine-grained type) pairs from two perspectives. The experiment results on the Twitter FMNERG dataset show that MCCDR is 6% to 10% higher than the baseline and achieves state-of-the-art results on the FewFMNER task.

Deep neural networks operating on non-Euclidean geometries have recently demonstrated impressive performance across various machine-learning applications. Several studies have extended the attention mechanism to different manifolds. However, most existing non-Euclidean attention models are tailored to specific geometries, limiting their applicability. On the other hand, recent studies show that several matrix manifolds, such as Symmetric Positive Definite (SPD), Symmetric Positive Semi-Definite (SPSD), and Grassmannian manifolds, admit gyrovector structures, which extend vector addition and scalar product into manifolds. Leveraging these properties, we propose a Gyro Attention (GyroAtt) framework over general gyrovector spaces, applicable to various matrix geometries. Empirically, we manifest GyroAtt on three gyro structures on the SPD manifold, three on the SPSD manifold, and one on the Grassmannian manifold. Extensive experiments on four electroencephalography (EEG) datasets demonstrate the effectiveness of our framework.

Existing multi-modal object tracking approaches primarily focus on dual-modal paradigms, such as RGB-Depth or RGB-Thermal, yet remain challenged in complex scenarios due to limited input modalities. To address this gap, this work introduces a novel multi-modal tracking task that leverages three complementary modalities, including visible RGB, Depth (D), and Thermal Infrared (TIR), aiming to enhance robustness in complex scenarios. To support this task, we construct a new multi-modal tracking dataset, coined RGBDT500, which consists of 500 videos with synchronised frames across the three modalities. Each frame provides spatially aligned RGB, depth, and thermal infrared images with precise object bounding box annotations.Furthermore, we propose a novel multi-modal tracker, dubbed RDTTrack.RDTTrack integrates tri-modal information for robust tracking by leveraging a pretrained RGB-only tracking model and prompt learning techniques.In specific, RDTTrack fuses thermal infrared and depth modalities under a proposed orthogonal projection constraint, then integrates them with RGB signals as prompts for the pre-trained foundation tracking model, effectively harmonising tri-modal complementary cues.The experimental results demonstrate the effectiveness and advantages of the proposed method, showing significant improvements over existing dual-modal approaches in terms of tracking accuracy and robustness in complex scenarios. The dataset and source code are publicly available at https://xuefeng-zhu5.github.io/RGBDT500.

Sketch-based 3D shape retrieval (SBSR) aims to retrieve 3D shapes using hand-drawn sketches as query inputs. Although existing SBSR methods have achieved promising results, several challenges still require further investigation. First, most existing approaches usually leverage simple aggregation schemes, often failing to capture the intrinsic relationships between views, which limits the effectiveness of 3D shape feature extraction. Second, conventional SBSR primarily focuses on instance-level alignment while ignoring multi-level alignment, which may neglect complex hierarchical relationships. To address these limitations, we propose a novel Multi-level Encoding with Hierarchical Alignment (MEHA) method for SBSR. Specifically, we adopt spatial encoding and view encoding for multiple views of 3D shapes. The proposed aggregation scheme then integrates these multi-level embedded local features to enhance the representation of 3D shape features. Considering the complexity of 3D shapes, MEHA adopts a two-stage training process: the first stage focuses on learning 3D shape features, while the second stage emphasizes modality alignment. Furthermore, we introduce a hierarchical alignment strategy that bridges the modality gap through instance-level, prototype-level, and centre-level alignment. Extensive experiments on two public benchmark datasets demonstrate the superiority of our method, showing that MEHA outperforms the state-of-the-art baselines.

Contrastive learning for deep multi-view clustering aims to learn discriminative representations across multiple views. However, prevailing cluster-level alignment approaches fail to fully leverage cross-view consistency and complementarity, as they neglect instance-level semantic coherence. To address this limitation, we propose a novel bidirectional contrastive learning network for multi-view clustering. By simultaneously contrasting the inter-view semantic label matrix along the row and column directions (i.e., at instance-level and cluster-level), the labels of the same instance in different views are consistent, and instances assigned to the same cluster across different views remain consistent. Moreover, we use dual-channel MLPs to avoid information conflicts caused by bidirectional contrastive learning. The proposed framework also demonstrates strong generalization capability, serving as a plug-and-play module that can be seamlessly integrated with existing methods to improve their clustering performance. Extensive experiments on publicly available datasets demonstrate the superiority of our method over several state-of-the-art techniques.

Prompt-tuning has shown remarkable success in multi-modal visual tracking, which enhances RGB tracking by incorporating an additional modality,e.g., thermal infrared (T), Depth (D), or Event (E), forming an RGB+X tracking paradigm. However, in the testing phase, current approaches utilise a frozen prompt for the entire benchmark, failing to account for the diversity and unique characteristics of individual videos. To address this issue, we inject meta-learning solutions into current prompt-based tracking technique, thereby emphasising sequence-level adaptation. Unlike traditional prompt-based trackers, which keep the parameters of prompt blocks fixed during testing, our approach updates these parameters in the first frame of each video using a meta-learning solution. This allows for enhanced discriminative tracking capabilities tailored to each video. Building on this advancement, we extend our methodology beyond separate implementations for RGBT, RGBD, and RGBE tasks. A unified multi-modal tracker is further derived, resulting in the first unified tracker without any task priors (notification of task type) employed in both training and testing phases. Extensive experimental results on LasHeR, DepthTrack, VisEvent, GTOT, RGBT234, and RGBD1K consistently demonstrate the superiority of the proposed method against the existing prompt-tuning paradigm. Source codes are available athttps://github.com/Zhangyong-Tang/M3Track.

RGB-Thermal (RGB-T) salient object detection aims to accurately locate salient regions by integrating complementary information from visible and thermal modalities. However, existing methods often struggle to fully leverage the spatial structures and semantic differences in the frequency domain across modalities, while also incurring high computational costs. In this paper, we propose a lightweight spatial-channel-frequency information mining framework (SCF-Net) for RGB-T salient object detection. Specifically, we first introduce a local-global complementary aggregation (LGCA) module that effectively fuses local textures, global semantics, and modality-aware features to enhance cross-layer information consistency and semantic alignment. Furthermore, we design a triple cue mining (TCM) module to jointly explore spatial, channel, and frequency-domain salient cues, enabling the extraction of both intra-modal details and cross-modal complementary information. Our SCF-Net contains only 8.39 M parameters and supports real-time inference. Extensive experiments conducted on VT821, VT1000, and VT5000 three datasets demonstrate that our method consistently outperforms existing mainstream models while maintaining high operational efficiency, showing strong effectiveness and robustness across a variety of challenging scenarios.

Recent advances in RGB-D tracking have been driven by the synergistic combination of high-performing RGB-only trackers and auxiliary depth information. However, most existing methods rely on visual feature descriptors to extract depth features, which are then fused with vision features. This pipeline may lead to performance degradation due to the incongruence between the RGB and depth modalities. In this letter, we propose an efficient and effective transformer-based framework, that explicitly models colour and depth information for RGB-D tracking. Specifically, we first statistically code the colour and depth information of the foreground and background for the template. Then, the spatial attention maps of the search region are obtained using these colour-depth statistical models, enhancing the visual features of the search region for improved object localisation accuracy. The comprehensive experimental results obtained on multiple benchmarks demonstrate the effectiveness and merits of the proposed approach in explicit colour-depth coding for RGB-D tracking. The code and the models are publicly accessible athttps://github.com/xuefeng-zhu5/CDAAT.

Clustering is a crucial step in single-cell RNA sequencing (scRNA-seq) data analysis, facilitating the discovery of new cell types and the grouping of similar cells. Recently, graph convolutional networks (GCNs) have gained prominence in scRNA-seq data clustering because they effectively learn cell representations by capturing the relationship between cells. However, GCNs are sensitive to noise in scRNA-seq data and are prone to over-smoothing, resulting in the loss of cell-specific information. To overcome these challenges, we propose sigRGCN, a robust residual graph convolutional network for scRNA-seq data clustering. Specifically, we first construct a disturbed cell graph by injecting noise into a cell graph constructed from scRNA-seq data. Then, we design a graph structure optimization graph convolutional network to eliminate the impact of noise in the disturbed cell graph. It significantly improves the robustness of the proposed model in real scRNA-seq data clustering tasks. After that, we utilize a $L$-layers residual graph convolutional network to alleviate the over-smoothing problem. It allows our model to effectively capture higher-order relationships between cells, leading to better cell representations. Finally, we employ a self-supervised manner to optimize our model. The experimental results on nine real scRNA-seq datasets show that our proposed model demonstrates competitive performance in real clustering tasks.

Multi-view clustering based on deep auto-encoder networks has garnered increasing attention and made significant progress in recent years. However, we argue that most existing methods inadequately explore the discriminability while learning clustering assignments, resulting in models struggling to accurately cluster data, particularly those with ambiguous semantics. To address this problem, we propose a novel framework termed deep discriminative multi-view clustering (DDMvC). This framework is designed to further increase the inter-cluster distances by learning a discriminative projection dictionary with global prior information. To begin with, we enhance the reliability of the dictionary atoms by initializing them with class-specific prototypes derived from concatenated global features across multiple views. Subsequently, we iteratively refine the atoms to guarantee their independence from any specific cluster. Simultaneously, we incorporate contrastive learning for the cluster assignments projected by these atoms, striving for inter-view consistent clustering results. Experimental results on benchmark multi-view datasets demonstrate that our framework achieves the state-of-the-art clustering performance.

In recent years, zero-shot sketch-based image retrieval (ZS-SBIR) task has attracted considerable attention. Although some ZS-SBIR approaches have been proposed, it remains challenging to handle the inherent linkages between the sketch and image domains. Moreover, how to transfer semantic knowledge from seen categories to unseen categories is still an open problem, significantly affecting retrieval performance. In this article, we propose a novel approach Modality Fused Class-Proxy with Knowledge Distillation, named MFCPKD, which develops two novel schemes to remedy the above issues. Specifically, MFCPKD leverages a Modality Fusion Model to learn modality-fused feature embeddings and class proxies. The knowledge distillation is employed for student to learn the feature from seen categories and infer the unknown category through class proxies. Furthermore, three losses constrain the student network to narrow the modality gap between sketch and image domains. Finally, we conduct extensive experiments on three benchmark datasets (Sketchy Ext, TU-Berlin Ext, and QuickDraw Ext) and demonstrate that our MFCPKD method can achieve excellent performance compared to some existing methods in ZS-SBIR scenarios. The code for our project is available athttps://github.com/li1changxing/MFCPKD.

Recent hyperspectral image (HSI) classification works have focused on developing a promising architecture by combining convolutional neural networks (CNNs) with Transformers. However, most of them fail to consider the interactive fusion of global and local features, thus significantly limiting the HSI classification performance. To address this problem, in this article, we propose a dual feature aggregation network (DFAN) for HSI classification tasks. It effectively aggregates local and global spatial-spectral features to achieve efficient classification. Specifically, we design a dual feature aggregation (DFA) module to extract and aggregate local and global spatial-spectral features. In this module, the lightweight local block is responsible for extracting local spatial-spectral features, and the global block is used to extract global spatial-spectral features and aggregation tasks. Concretely, the token local aggregation multilayer perceptron (TLA-MLP) module in the global block is to extract spatial-position-aware and spectral-channel-aware global discriminative information. Meanwhile, it learns multiscale neighboring token features by aggregating token local neighborhood features. In addition, we employ the local self-global aggregation block to learn the global aggregation features and then fuse them with the local spatial-spectral features. Afterward, the cross-attention aggregation of local and global spatial-spectral features is used to further improve the classification ability of our proposed model. The experimental results on three benchmark datasets show that our proposed DFAN method outperforms other state-of-the-art HSI classification methods.

Vision Transformers (ViTs) have achieved remarkable performance in computer vision tasks but are vulnerable to adversarial attacks. Recent studies have demonstrated the feasibility of crafting transferable adversarial examples based on ViT models. However, the adversarial examples generated by ViTs exhibit poor generalization, primarily due to structural differences between models and the tendency to overfit, which significantly hinders cross-architecture transferability. In this paper, we propose a novel framework to improve the generalization and transferability of adversarial attacks across diverse models, focusing on two key strategies: Token Gradient Divergence (TGD) and Multi-level Frequency-aware Attack (MFA). TGD, as a gradient regularization method, addresses the structural gradient issue of surrogate models, which is one of the causes of overfitting. By increasing the gradient divergence between tokens and eliminating the influence of the class token gradient, TGD enhances the transferability of adversarial examples across models. Meanwhile, MFA employs an implicit ensemble approach to enhance attack generalization. Through multiple spectral augmentations, it increases input diversity and simulates ensemble learning. By targeting critical frequency regions across models, MFA enhances adversarial example adaptability to different architectures, significantly boosting cross-architecture transferability. Extensive experiments on both ViTs and CNNs demonstrate that TGD-MFA significantly outperforms state-of-the-art transfer-based attacks, achieving substantial improvements in adversarial transferability and robustness.

The widespread application of deep learning-based face recognition (FR) systems poses significant challenges to the privacy of facial images on social media, as unauthorized FR systems can exploit these images to mine user data. Recent studies have utilized adversarial attack techniques to protect facial privacy against malicious FR systems by generating adversarial examples. However, existing noise-based and makeup-based methods produce adversarial examples with noticeable noise or undesired makeup attributes, and suffers from low transferability issues. In this paper, we propose a novel stealthy-based approach, named Dual-latent Adaptive Diffusion Protection (DADP), which generates transferable stealthy adversarial examples consistent with the source images by the diffusion model to protect facial privacy. DADP effectively harnesses adversarial information within both the semantic and diffusion latent spaces to explore adversarial latent representations. Unlike traditional methods that rely on bounded constraints and sign gradient optimization, DADP employs adaptive optimization to maximize the utilization of adversarial gradient information and introduces latent regularization to constrain the adaptive optimization process, ensuring that the protected faces maintain high privacy and natural appearance. Extensive qualitative and quantitative experiments on the public CelebA-HQ and LADN datasets demonstrate the proposed method crafts more natural-looking stealthy adversarial examples with superior black-box transferability compared to the state-of-the-art methods. The code is released at https://github.com/LiYuanBoJNU/DADP.

Semi-supervised learning has gained considerable popularity in medical image segmentation tasks due to its capability to reduce reliance on expert-examined annotations. Several mean-teacher (MT) based semi-supervised methods utilize consistency regularization to effectively leverage valuable information from unlabeled data. However, these methods often heavily rely on the student model and overlook the potential impact of cognitive biases within the model. Furthermore, some methods employ co-training using pseudo-labels derived from different inputs, yet generating high-confidence pseudo-labels from perturbed inputs during training remains a significant challenge. In this paper, we propose an Uncertainty-aware Cross-training framework for semi-supervised medical image Segmentation (UC-Seg). Our UC-Seg framework incorporates two distinct subnets to effectively explore and leverage the correlation between them, thereby mitigating cognitive biases within the model. Specifically, we present a Cross-subnet Consistency Preservation (CCP) strategy to enhance feature representation capability and ensure feature consistency across the two subnets. This strategy enables each subnet to correct its own biases and learn shared semantics from both labeled and unlabeled data. Additionally, we propose an Uncertainty-aware Pseudo-label Generation (UPG) component that leverages segmentation results and corresponding uncertainty maps from both subnets to generate high-confidence pseudo-labels. We extensively evaluate the proposed UC-Seg on various medical image segmentation tasks involving different modality images, such as MRI, CT, ultrasound, colonoscopy, and so on. The results demonstrate that our method achieves superior segmentation accuracy and generalization performance compared to other state-of-the-art semi-supervised methods. Our code and segmentation maps will be released at https://github.com/taozh2017/UCSeg.

The preservation and the enhancement of complementary features between modalities are crucial for multi-modal image fusion and downstream vision tasks. However, existing methods are limited to local receptive fields (CNNs) or lack comprehensive utilization of spatial information from both modalities during interaction (transformers), which results in the inability to effectively retain useful information from both modalities in a comparative manner. Consequently, the fused images may exhibit a bias towards one modality, failing to adaptively preserve salient targets from all sources. Thus, a novel fusion framework (S4Fusion) based on the Saliency-aware Selective State Space is proposed. S4Fusion introduces the Cross-Modal Spatial Awareness Module (CMSA), which is designed to simultaneously capture global spatial information from all input modalities and promote effective cross-modal interaction. This enables a more comprehensive representation of complementary features. Furthermore, to guide the model in adaptively preserving salient objects, we propose a novel perception-enhanced loss function. This loss aims to enhance the retention of salient features by minimizing ambiguity or uncertainty, as measured at a pre-trained model's decision layer, within the fused images. The code is available at https://github.com/zipper112/S4Fusion.

In recent years, deep learning-based methods have made significant progress on the image quality assessment problem; however, challenges remain arising from the lack of annotated, real-world training data and consequent poor generalization ability. Towards addressing these challenges, we propose a no-reference image quality assessment (NR-IQA) method based on generative AI (GenAI) images. Specifically, we use GenAI images as reference images, employing a cold diffusion model to generate distorted images of four different distortion types, and we label these distorted images using a full-reference model, thereby making it possible to construct a large-scale pre-training dataset. We use this resource generation method to facilitate NR-IQA model building. We deploy a Multi-scale Cross Attention Block (MCAB) and a Scale Simple Attention Module (SSAM) to enhance feature representation by extracting multi-scale feature information from both the channel and spatial dimensions that are predictive of image quality. Extensive experiments on eight public databases demonstrate that the proposed method achieves state-of-the-art (SOTA) performance. A public release of all the codes associated with this work will be made available on GitHub.

RGBT tracking draws increasing attention because of its robustness in multi-modal warranting (MMW) scenarios, such as nighttime and adverse weather conditions, where relying on a single sensing modality fails to ensure stable tracking results. However, existing benchmarks predominantly contain videos collected in common scenarios where both RGB and thermal infrared (TIR) information are of sufficient quality. This weakens the representativeness of existing benchmarks in severe imaging conditions, leading to tracking failures in MMW scenarios. To bridge this gap, we present a new benchmark considering the modality validity, MV-RGBT, captured specifically from MMW scenarios where either RGB (extreme illumination) or TIR (thermal truncation) modality is invalid. Hence, it is further divided into two subsets according to the valid modality, offering a new compositional perspective for evaluation and providing valuable insights for future designs. Moreover, MV-RGBT is the most diverse benchmark of its kind, featuring 36 different object categories captured across 19 distinct scenes. Furthermore, considering severe imaging conditions in MMW scenarios, a new problem is posed in RGBT tracking, named 'when to fuse', to stimulate the development of fusion strategies for such scenarios. To facilitate its discussion, we propose a new solution with a mixture of experts, named MoETrack, where each expert generates independent tracking results along with a confidence score. Extensive results demonstrate the significant potential of MV-RGBT in advancing RGBT tracking and elicit the conclusion that fusion is not always beneficial, especially in MMW scenarios. Besides, MoETrack achieves state-of-the-art results on several benchmarks, including MV-RGBT, GTOT, and LasHeR. Source codes and benchmarks are available at https://github.com/Zhangyong-Tang/MVRGBT.

Multi-view clustering aims at partitioning data into their underlying categories by mining shared and complementary information conveyed by different views. Although the integration of deep learning and disentanglement learning has markedly improved clustering performance, our analysis reveals two fundamental limitations in existing approaches: inadequate separation between view-shared and view-exclusive features; and the negative effects of clustering-irrelevant information on feature decoupling. To tackle these issues, we present a novel Disentangled Feature Learning Network (DFL-Net), which utilizes a progressive learning framework to systematically disentangle features. DFL-Net initially establishes view-shared representations through semantic disparity minimization, followed by the construction of orthogonal feature subspaces using cross-view and intra-view independence constraints to isolate view-specific features. Subsequently, DFL-Net enforces clustering consistency across views to adaptively eliminate irrelevant information, thus enhancing the overall effectiveness of disentanglement learning. The framework introduces two significant innovations: a comprehensive feature independence criterion that concurrently reduces intra-view and cross-view feature dependencies, and an irrelevance filtering mechanism that ensures cross-view clustering consistency. Extensive experiments on benchmark datasets demonstrate the superior performance of DFL-Net compared to state-of-the-art methods.

Predicting where people look is a crucial for understanding human intentions. Gaze prediction, as a research hotspot, has evolved from predicting the gaze of a single person to simultaneously predicting the positions of all individuals and their corresponding gaze targets. However, the study of the correlation between humans and gaze as two interdependent tasks has largely been neglected. In this paper, inspired by the concept of “mutualistic symbiosis” in ecology, we propose a novel multitask mutualistic transformer (MMTR). MMTR captures paired dependencies by establishing information communication between different branches, thereby enabling comprehensive and interpretable gaze analysis for all individuals and gaze targets. Specifically, we first utilize a transformer encoder to capture the common features of all the tasks. Then, we design a mutualistic attention mechanism (MAM) in the dual-branch Transformer decoder to establish cross-task information interaction. The MAM can learn privileged information from other tasks that is helpful for the current task, thereby guiding the current branch to learn the most valuable and distinctive features. To the best of our knowledge, this is the first time that privileged information has been introduced into the gaze estimation task. Furthermore, to more flexibly learn pixel locality and long-range semantic dependencies for different tasks, we construct and embed a learnable global-local position encoding (GLPE) in different branches of MMTR. Experiments demonstrate that our proposed MMTR can guide the two branches to communicate through privileged information, effectively solve the information asymmetry problem between human detection and gaze prediction, and significantly outperform state-of-the-art gaze prediction methods on two standard benchmark datasets GazeFollowing and VideoAttentionTarget.

Enabled by hierarchical convolutions and nonlinear mappings, recent action recognition studies have continuously boosted performance with spatiotemporal modelling. In general, motion clues are essential in video-oriented tasks, while existing approaches aggregate the spatial and temporal signatures via specially designed modules in the middle or output stages. To highlight the privilege provided by temporal motions, in this paper, we propose a simple but effectiveMOTion Estimator(MOTE) to generate the motion patterns from every single frame, avoiding complex dense-frame input. In particular, MOTE follows an encoder-decoder structure, which takes the short-term motion features generated by the pretrained dense-frame network as the learning target. The spatial information of a single frame is utilized to estimate the instantaneous motion appearance. It can support the expression of vulnerable regions, such as the ‘hand’ in ‘waving hands’, which would otherwise be suppressed in the feature maps as the ‘hand’ suffers from motion blur. The training process of MOTE is independent of the action recognition system. Therefore, the trained MOTE can be transplanted to the input-end of existing action recognition methods to provide instantaneous motion estimation as feature enhancement according to practical requirements. Our experiments performed on Something-Something V1, V2, Kinetics-400, and Diving48 verify the effectiveness of the proposed method.

Symmetric positive definite (SPD) matrices have been widely used as Riemannian feature descriptors in various scientific fields, due to their capacity to encode effective manifold-valued representations. Inspired by the architectural principles of Euclidean deep learning, the emerging SPD neural networks have achieved more robust signal classification. Among these advancements, Riemannian batch normalization (RBN) based on the affine-invariant Riemannian metric (AIRM) has emerged as a key technique for enhancing the learning capability of SPD-based networks. Nevertheless, the reliance of singular value decomposition (SVD) makes this metric relatively unstable for the computation of SPD matrices, especially for the ill-conditioned case. To address this limitation, we propose a novel RBN algorithm based on the recently introduced log-Cholesky metric (LCM), which leverages Cholesky decomposition. Unlike AIRM, the LCM offers enhanced numerical stability and allows for more efficient computation. Specifically, the LCM-based Riemannian operators such as Fr $\acute {\mathrm {e}}$ chet mean and parallel transport (PT) are much simpler than those of AIRM, and both have closed forms. Besides, since LCM is the pullback metric from the Cholesky manifold via Cholesky decomposition, the LCM-based RBN on the SPD manifold can be computed in the Cholesky manifold, further boosting the efficiency. Extensive experiments conducted on four benchmarking datasets certify the effectiveness of our proposed algorithm. The source code is now available at: https://github.com/jjscc/CBN.git.

RGB-Thermal semantic segmentation (SS) aims to combine visual light and thermal images to determine the semantic category for each pixel and create an object mask. While existing methods typically rely on well-aligned RGB-T image pairs, real-world RGB-T pairs are often unaligned, and pixel-by-pixel alignment is both challenging and time-consuming. To address this critical issue, we introduce a new unaligned RGB-T SS benchmark and propose the deformation-resilient multigranularity learning (DML) method. DML explores the spatial consistency and modal complementarity of RGB-T and mitigates the interference of warped modalities by aligning multimodal features in a coarse-to-fine multigranularity strategy. Specifically, DML constructs a deformation-aware complementary feature enhancer (DCFE), which consists of deformation-aware feature alignment (DFA) and complementary feature aggregation (CFA) modules. DFA enhances the spatial alignment of RGB-T by estimating the deformation field of warped features. Then, CFA aggregates complementary contexts of modal differences across multiple scales to produce deformation-resilient and robust RGB-T feature representations. Finally, we design the multigranularity mask refinement engine (MMFE), which combines class-agnostic saliency prediction (CSP) and class-aware edge generation (CEG) auxiliary tasks to provide useful boundary and positional cues for SS decoders. The MMFE enhances semantic alignment and interclass separability, yielding object masks with sharp boundaries. Quantitative and qualitative experiments on aligned and unaligned datasets validate the effectiveness of our proposed DML, consistently outperforming existing methods designed for aligned RGB-T data. The new unaligned RGB-T SS benchmark and code are available at https://github.com/VisionVerse/Unaligned-RGBT-Semantic-Segmentation.

Image restoration (IR) involves the retrieval of missing or damaged image information and represents a significant challenge in the field of visual reconstruction. Currently, U-Net based Diffusion Models (DMs) display favorable results when utilized for IR tasks. However, the DM based on U-Net demonstrates shortcomings in capturing the global context for IR. To address this issue, we propose a Novel Image Restoration Approach Based on U-shaped Transformer and DMs (DIRformer). DIRformer enhances the modeling capacity for long-range dependencies within DMs. In particular, DIRformer replaces the traditional U-Net downsampling with Patch merging, dedicated to improving detail preservation, and replaces upsampling with Dual up-sample, strategically designed to alleviate checkerboard artifacts. Besides, as a lightweight and versatile transformer-based solution for IR, DIRformer incorporates time and degradation mapping into the transformer design, all while preserving the fundamental U-shaped structural framework. We assess the efficacy of DIRformer in a multi-tasking IR setting across four datasets. The experimental performance illustrates that DIRformer achieves competitive performance on distortion metrics, including PSNR and SSIM. Remarkably, our proposed approach is almost 25 \(\times\) smaller and 2 \(\times\) faster than the existing methods while achieving comparable high performance.

In recent years, pretrained language-image models (PLIMs) have delivered advances in video captioning. However, existing PLIMs primarily focus on extracting global feature representations from still images and text sequences, while neglecting fine-grained semantic alignment and temporal variations between vision and text pairs. To this end, we propose a global-local alignment module and a temporal parsing module to reflect the detailed correspondence and temporal perception between the two modalities, respectively. In particular, the global-local alignment module enables cross-modal registration at two levels, i.e., the sentence-video level and the word-frame level, to obtain mixed-granularity semantic video features. The temporal parsing module is a dedicated self-attention structure that highlights temporal order cues along video frames, compensating for the limited temporal capacity of PLIMs. In addition, an adaptive two-stage gating structure is designed to leverage the linguistic predictions further. The linguistic information derived from the first stage prediction is dynamically routed through an adaptive decision gate, allowing for quality assessment of whether the information should proceed to the second stage. This structure can effectively reduce the computational burden for easy samples and further improve the accuracy of the prediction results. The experimental results obtained on several benchmark datasets demonstrate the effectiveness of the proposed solution, with improved performance compared to state-of-the-art methods.

Generative models (GMs) have received increasing research interest for their remarkable capacity to achieve comprehensive understanding. However, their potential application in the domain of multi-modal tracking has remained unexplored. In this context, we seek to uncover the potential of harnessing generative techniques to address the critical challenge, information fusion, in multi-modal tracking. In this paper, we delve into two prominent GM techniques, namely, Conditional Generative Adversarial Networks (CGANs) and Diffusion Models (DMs). Different from the standard fusion process where the features from each modality are directly fed into the fusion block, we combine these multi-modal features with random noise in the GM framework, effectively transforming the original training samples into harder instances. This design excels at extracting discriminative clues from the features, enhancing the ultimate tracking performance. Based on this, we conduct extensive experiments across two multi-modal tracking tasks, three baseline methods, and four challenging benchmarks. The experimental results demonstrate that the proposed generative-based fusion mechanism achieves state-of-the-art performance by setting new records on GTOT, LasHeR and RGBD1K. Code will be available at https://github.com/Zhangyong-Tang/GMMT.

Contrastive learning has achieved great success in skeleton-based action recognition. However, most existing approaches encode the skeleton sequences as entangled spatiotemporal representations and confine the contrasts to the same level of representation. Instead, this paper introduces a novel contrastive learning framework, namely Spatiotemporal Clues Disentanglement Network (SCD-Net). Specifically, we integrate the decoupling module with a feature extractor to derive explicit clues from spatial and temporal domains respectively. As for the training of SCD-Net, with a constructed global anchor, we encourage the interaction between the anchor and extracted clues. Further, we propose a new masking strategy with structural constraints to strengthen the contextual associations, leveraging the latest development from masked image modelling into the proposed SCD-Net. We conduct extensive evaluations on the NTU-RGB+D (60&120) and PKU-MMD (I&II) datasets, covering various downstream tasks such as action recognition, action retrieval, transfer learning, and semi-supervised learning. The experimental results demonstrate the effectiveness of our method, which outperforms the existing state-of-the-art (SOTA) approaches significantly. Our code and supplementary material can be found at https://github.com/cong-wu/SCD-Net.

Deep neural networks for learning Symmetric Positive Definite (SPD) matrices are gaining increasing attention in machine learning. Despite the significant progress, most existing SPD networks use traditional Euclidean classifiers on an approximated space rather than intrinsic classifiers that accurately capture the geometry of SPD manifolds. In-spired by Hyperbolic Neural Networks (HNNs), we propose Riemannian Multinomial Logistics Regression (RMLR) for the classification layers in SPD networks. We introduce a unified framework for building Riemannian classifiers under the metrics pulled back from the Euclidean space, and showcase our framework under the parameterized Log-Euclidean Metric (LEM) and Log-Cholesky Metric (LCM). Besides, our framework offers a novel intrinsic explanation for the most popular LogEig classifier in existing SPD networks. The effectiveness of our method is demonstrated in three applications: radar recognition, human action recognition, and electroencephalography (EEG) classification. The code is available at https://github.com/GitZH-Chen/SPDMLR.git.

Heuristic algorithms have been developed to find approximate solutions for high-utility itemset mining (HUIM) problems that compensate for the performance bottlenecks of exact algorithms. However, heuristic algorithms still face the problem of long runtime and insufficient mining quality, especially for large transaction datasets with thousands to tens of thousands of items and up to millions of transactions. To solve these problems, a novel GPU-based efficient parallel heuristic algorithm for HUIM (PHA-HUIM) is proposed in this paper. The iterative process of PHA-HUIM consists of three main steps: the search strategy, fitness evaluation, and ring topology communication. The search strategy and ring topology communication are designed to run in constant time on GPU. The parallelism of fitness evolution helps to substantially accelerate the algorithm. To improve the mining quality, a multi-start strategy with an unbalanced allocation strategy is employed in the search process. Ring topology communication is adopted to maintain population diversity. A load balancing strategy is introduced to reduce the thread divergence to improve the parallel efficiency. The experimental results on nine large datasets show that PHA-HUIM outperforms state-of-the-art HUIM algorithms in terms of speedup performance, runtime, and mining quality.

Bayesian network structure learning (BNSL) from data is an NP-hard problem. Genetic algorithms are powerful for solving combinatorial optimization problems, but the lack of effective guidance results in slow convergence and low accuracy regarding BNSL. To address this problem, we propose a mutual information (MI) guided genetic algorithm (MIGA) for BNSL in this paper, which uses MI to effectively search BN structures. In the initialization phase of MIGA, the population is generated by adding additional constraints based on MI to reach a higher score without losing diversity. By employing normalized MI and defining the population support, the potential dominance in the population can be identified and then used to design a novel crossover operator in order to preserve the dominant genes with a higher probability. Moreover, with the guidance of MI for removing loops from the structures, infeasible solutions can be handled in a straightforward and practical way. The proposed MIGA is evaluated on eleven well-known benchmark datasets and compared with four GA-based methods and four other state-of-the-art BNSL algorithms. Experimental results show that MIGA outperforms the compared algorithms in convergence and learning accuracy.

The generalization of Euclidean network paradigm to the Riemannian manifolds has attracted much attention for offering useful geometric representations in processing manifold-valued data in recent years. However, the information degradation during data compression mapping hinders Riemannian networks from going deeper, and there are very few solutions specifically designed for this problem. Given the remarkable success of deep Residual learning in Euclidean networks, a novel Riemannian residual learning mechanism (RRLM) is proposed in the context of Symmetric Positive Definite (SPD) manifolds, enabling the characterization of deep spatiotemporal features while preserving the manifold properties. Based on RRLM, a stack of SPD manifold-constrained residual-like blocks is designed on the tail of the original SPDNet(backbone) for the sake of conducting deep Riemannian residual learning. For simplicity, we refer to the network architecture introduced above as Riemannian residual SPD network (ResSPDNet). The experimental results achieved on three types of visual classification tasks, i.e., facial emotion recognition, drone recognition, and action recognition, demonstrate that our method can achieve improved accuracy with a deepened network structure.

Recently, depth ambiguity reduction has made significant progress in monocular human mesh reconstruction. However, regression-based methods struggle to perceive minor depth deviations due to insufficiently exploited monocular cues, such as linear perspective, shadows, and pose priors of the given image. To tackle this limitation, we propose a Depth-aware Updated Feature Feedback (DAUFF) loop to iteratively exploit spatial information from rendered images. Specifically, the normalized depth and dense correspondence images rendered from the ground truth human meshes are adopted as auxiliary supervision for more reliable and adequate cues. Moreover, we also use dense correspondences for feature update to improve the adaptability to new predictions in the loop. Remarkably, dense correspondences can facilitate the rectification of parameters by providing a fine-grained perception of the current prediction. Extensive quantitative comparison on standard benchmarks shows that DAUFF achieves better-aligned reconstruction results than existing approaches. The dataset and codes will be available on the project homepage: https://github.com/github1838/DAUFF.

In recent years, Multi-Modality Image Fusion (MMIF) has been applied to many fields, which has attracted many scholars to endeavour to improve the fusion performance. However, the prevailing focus has predominantly been on the architecture design, rather than the training strategies. As a low-level vision task, image fusion is supposed to quickly deliver output images for observation and supporting downstream tasks. Thus, superfluous computational and storage overheads should be avoided. In this work, a lightweight Distilled Mini-Model with a Dynamic Refresh strategy (MMDRFuse) is proposed to achieve this objective. To pursue model parsimony, an extremely small convolutional network with a total of 113 trainable parameters (0.44 KB) is obtained by three carefully designed supervisions. First, digestible distillation is constructed by emphasising external spatial feature consistency, delivering soft supervision with balanced details and saliency for the target network. Second, we develop a comprehensive loss to balance the pixel, gradient, and perception clues from the source images. Third, an innovative dynamic refresh training strategy is used to collaborate history parameters and current supervision during training, together with an adaptive adjust function to optimise the fusion network. Extensive experiments on several public datasets demonstrate that our method exhibits promising advantages in terms of model efficiency and complexity, with superior performance in multiple image fusion tasks and downstream pedestrian detection application. The code of this work is publicly available at https://github.com/yanglinDeng/MMDRFuse.

Riemannian neural networks, which extend deep learning techniques to Riemannian spaces, have gained significant attention in machine learning. To better classify the manifold-valued features, researchers have started extending Euclidean multinomial logistic regression (MLR) into Riemannian manifolds. However, existing approaches suffer from limited applicability due to their strong reliance on specific geometric properties. This paper proposes a framework for designing Riemannian MLR over general geometries, referred to as RMLR. Our framework only requires minimal geometric properties, thus exhibiting broad applicability and enabling its use with a wide range of geometries. Specifically, we showcase our framework on the Symmetric Positive Definite (SPD) manifold and special orthogonal group, i.e., the set of rotation matrices. On the SPD manifold, we develop five families of SPD MLRs under five types of power-deformed metrics. On rotation matrices we propose Lie MLR based on the popular bi-invariant metric. Extensive experiments on different Riemannian backbone networks validate the effectiveness of our framework.

As a list of remotely sensed data sources is available, the effective processing of remote sensing images is of great significance in practical applications in various fields. This paper proposes a new lightweight network to solve the problem of remote sensing image processing by using the method of deep learning. Specifically, the proposed model employs ShuffleNet V2 as the backbone network, appropriately increases part of the convolution kernels to improve the classification accuracy of the network, and uses the maximum overlapping pooling layer to enhance the detailed features of the input images. Finally, Squeeze and Excitation (SE) blocks are introduced as the attention mechanism to improve the architecture of the network. Experimental results based on several multisource data show that our proposed network model has a good classification effect on the test samples and can achieve more excellent classification performance than some existing methods, with an accuracy of 91%, and can be used for the classification of remote sensing images. Our model not only has high accuracy but also has faster training speed compared with large networks and can greatly reduce computation costs. The demo code of our proposed method will be available at https://github.com/li-zi-qi.

The task of Open-World Compositional Zero-Shot Learning (OW-CZSL) is to recognize novel state-object compositions in images from all possible compositions, where the novel compositions are absent during the training stage. The performance of conventional methods degrades significantly due to the large cardinality of possible compositions. Some recent works consider simple primitives (i.e., states and objects) independent and separately predict them to reduce cardinality. However, it ignores the heavy dependence between states, objects, and compositions. In this paper, we model the dependence via feasibility and contextuality. Feasibility-dependence refers to the unequal feasibility of compositions, e.g., hairy is more feasible with cat than with building in the real world. Contextuality-dependence represents the contextual variance in images, e.g., cat shows diverse appearances when it is dry or wet. We design Semantic Attention (SA) to capture the feasibility semantics to alleviate impossible predictions, driven by the visual similarity between simple primitives. We also propose a generative Knowledge Disentanglement (KD) to disentangle images into unbiased representations, easing the contextual bias. Moreover, we complement the independent compositional probability model with the learned feasibility and contextuality compatibly. In the experiments, we demonstrate our superior or competitive performance, SA-and-kD-guided Simple Primitives (SAD-SP), on three benchmark datasets.

Abstract Vulnerability classification is a significant activity in software development and software maintenance. Natural Language Processing (NLP) techniques, which utilize the descriptions in public repositories, are widely used in automatic software vulnerability classification. However, vulnerability descriptions are ordinarily short and contain many technical terms, making them difficult for machines to automatically comprehend. In this paper, we present an approach based on vulnerability triggers to automatically classify vulnerabilities. First, we extract vulnerability triggers with Bert Question and Answer (Bert Q&A). Then, we use Recurrent Convolutional Neural Networks for Text classification (TextRCNN) to classify vulnerabilities based on Common Weakness Enumeration (CWE). We statistically perform an analysis of vulnerability triggers and comprehensively evaluate the classification performance of our approach on a set of 4769 prelabeled vulnerability entries, as well as compare it with state‐of‐the‐art vulnerability classification approaches. Experiment results show that our approach can achieve a F1‐measure of 95% on extraction and 80.8% on classification.

Referring expression generation (REG) is a challenging task at the intersection of computer vision and natural language processing, which aims at generating natural language descriptions that uniquely refer to a specific object within an image. Existing REG approaches solely utilize bounding boxes in a rather primitive manner to specify target objects, and employ the classical Convolutional Neural Networks (CNNs) for image encoding, followed by recurrent layers for text generation. In this letter, we propose a novel end-to-end REG model. Our model highlights the target using bounding boxes and segmentations in a unified fashion. Specifically, we propose two settings for utilizing these signals: employing them as inputs to the model and as supervision signals for pre-training tasks. Additionally, we harness the power of the recently prevailed self-attention architecture to bridge targeted visual clues and text correspondence. During inference, our method achieves state-of-the-art performance in a one-stage manner, reflecting the potential of both bounding boxes and segmentation references in constructing REG solutions.

Cross-modal hashing retrieval approaches have received extensive attention owing to their storage superiority and retrieval efficiency. To achieve better retrieval performances, hashing methods seek to embed more semantic information of multi-modal data into hash codes. Existing deep cross-modal hashing methods typically learn hash functions from the similarity of paired data to generate hash codes. However, such locally-oriented learning methods often suffer from low efficiency and incomplete acquisition of semantic information. To address these challenges, this paper presents a novel deep hashing approach, called Proxy-based Graph Convolutional Hashing (PGCH), for cross-modal retrieval. Specifically, we use global similarity to construct proxy hash codes for two different modalities. This strategy of these proxy hash codes ensures that they include data points with significant distribution differences. It helps to match data from different modalities to different proxy hash codes, which can capture the global similarity of multi-modal hash codes and improve the efficiency of hash code learning. Subsequently, we employ a multi-modal contrastive loss to learn the global similarity. Furthermore, by constructing a proxy hash matrix from the proxy hash codes, we apply graph convolution to efficiently narrow the gap between different modalities, leading to a substantial improvement in retrieval performance for cross-modal retrieval tasks. The comprehensive experiments on four benchmark multimedia datasets demonstrate that our PGCH approach achieves better retrieval performances than a bundle of state-of-the-art hashing approaches.

Recently, mask-fill-based 3D Molecular Generation (MG) methods have become very popular in virtual drug design. However, the existing MG methods ignore the chemical properties of atoms and contain inappropriate atomic position training data, which limits their generation capability. To mitigate the above issues, this paper presents a novel mask-fill-based 3D molecule generation model driven by atomic chemical properties (APMG). Specifically, we construct a new attention-MPNN-based encoder and introduce the electronic information into atom representations to enrich chemical properties. Also, a multi-functional classifier is designed to predict the electronic information of each generated atom, guiding the type prediction of elements and bonds. By design, the proposed method uses the chemical properties of atoms and their correlations for high-quality molecule generation. Second, to optimize the atomic position training data, we propose a novel atomic training position generation approach using the Chi-Square distribution. We evaluate our APMG method on the CrossDocked dataset and visualize the docking states of the pockets and generated molecules. The obtained results demonstrate the superiority and merits of APMG over the state-of-the-art approaches.

Thanks to the efficacy of Symmetric Positive Definite (SPD) manifold in characterizing video sequences (image sets), image set-based visual classification has made remarkable progress. However, the issue of large intra-class diversity and inter-class similarity is still an open challenge for the research community. Although several recent studies have alleviated the above issue by constructing Riemannian neural networks for SPD matrix nonlinear processing, the degradation of structural information during multi-stage feature transformation impedes them from going deeper. Besides, a single cross-entropy loss is insufficient for discriminative learning as it neglects the peculiarities of data distribution. To this end, this paper develops a novel framework for image set classification. Specifically, we first choose a mainstream neural network built on the SPD manifold (SPDNet)[25]as the backbone with a stacked SPD manifold autoencoder (SSMAE) built on the tail to enrich the structured representations. Due to the associated reconstruction error terms, the embedding mechanism of both SSMAE and each SPD manifold autoencoder (SMAE) forms an approximate identity mapping, simplifying the training of the suggested deeper network. Then, the ReCov layer is introduced with a nonlinear function for the constructed architecture to narrow the discrepancy of the intra-class distributions from the perspective of regularizing the local statistical information of the SPD data. Afterward, two progressive metric learning stages are coupled with the proposed SSMAE to explicitly capture, encode, and analyze the geometric distributions of the generated deep representations during training. In consequence, not only a more powerful Riemannian network embedding but also effective classifiers can be obtained. Finally, a simple maximum voting strategy is applied to the outputs of the learned multiple classifiers for classification. The proposed model is evaluated on three typical visual classification tasks using widely adopted benchmarking datasets. Extensive experiments show its superiority over the state of the arts.

Modeling sequences with spatial-temporal graph convolutional networks has become a mainstream paradigm in skeleton-based action recognition. However, many existing methods adopt redundant or cluttered structures to mine the key action features, thus making it difficult to achieve a balanced or leading performance in accuracy and efficiency. In this paper, we propose a novel framework, referred to as Motion Complement and Temporal Multifocusing Network (MCTM-Net), to capture the relationships within skeleton sequences by means of an efficient decomposition of the spatiotemporal graph model. Specifically, for spatial modeling, we introduce a motion-related relational descriptor that extends the channel dimension so as to enhance the modeling of motion salient regions as a complement to the conventional physical adjacency relationships. An improved parameterized physical relationship model is also proposed to better fit the data characteristics. As for temporal modeling, we propose an efficient multi-focus temporal information acquisition strategy that aggregates the information from multiple temporal spans and adjacent regions. We conduct extensive experiments on multiple representative datasets, including NTU-RGB+D (60&120), Northwestern-UCLA, and UWA3D Multiview Activity II, to validate our innovations. The experimental results show the effectiveness of our method. The code will be available athttps://github.com/cong-wu/MCMT-Net.

Visual object tracking has witnessed continuous improvements in performance, thanks to deep CNN learning that recently emerged. More complex CNN models invariably offer better accuracy. However, there is a conflict between the tracking efficiency and model complexity, which poses a challenge in balancing speed against accuracy. To optimize the trade-off between these two performance criteria, a distillation-ensemble-selection framework is proposed in this paper. Without any modification to the baseline network architecture, the proposed approach enables the construction of a Siamese-based tracker with improved capacity and efficiency. Specifically, multiple student trackers are designed by means of knowledge distillation from a given teacher tracking model. To manage the varying granularity of unknown targets, an ensemble module combines the outputs of the student trackers with the help of a learnable fine-grained attention module. Besides, in the online tracking stage, a selection module adaptively controls the complexity of the tracker by identifying an appropriate subset of the candidate tracker models. We verify the effectiveness of the proposed method in both anchor-based and anchor-free paradigms. The experimental results obtained on standard benchmarking datasets demonstrate the effectiveness of the proposed method, with an outstanding and balanced performance in both accuracy and speed.

Harmful algal bloom (HAB) has long been one of the most formidable environmental problems in the world. HAB is influenced by multifactors, and its dynamic is highly nonstationary, making its prediction challenging. The existing machine learning (ML)-based HAB prediction methods mainly use time-series data, which ignore the intrinsic relationship between spatial and temporal variations in HAB. To achieve more accurate HAB spatiotemporal prediction, a novel long short-term memory (LSTM)-based nonstationary focusing prediction model (HabLSTM) is proposed in this article. The HabLSTM network is constructed by stacking HabLSTM units consisting of the hidden states spatial differential block (HSSD) and the combined states temporal differential (CSTD) block. The HSSD block uses the gating mechanism and the difference in hidden states to generate differential features between adjacent frames and guides the network to learn short-term nonstationary features by controlling the feature update of the hidden state in the HabLSTM unit. The CSTD block uses the gating mechanism and the difference in combined states to generate the differential features of the current input sequence and guides the network to learn long-term nonstationary features by controlling the feature update of the memory state in the HabLSTM unit. These two differential features guide the HabLSTM network to focus on learning nonstationary spatiotemporal features and boost HAB spatiotemporal prediction accuracy. In addition, two new spatiotemporal datasets of HAB named as Taihu HAB A and Taihu HAB B are established using the year-A and year-B normalized difference vegetation index (NDVI) images collected by Himawari-8 satellite, respectively. The experimental results on the two HAB datasets and spatiotemporal predictive learning (ST-PL) benchmark dataset MovingMNIST++ validate the outstanding HAB prediction and nonstationary spatiotemporal features’ learning capability of HabLSTM. The source code is available athttps://github.com/lxq-jnu/HabLSTM.

The Surface Water and Ocean Topography (SWOT) mission was recommended by the 2007 National Research Council Decadal Survey to expand on previous altimetry missions like TOPEX/Poseidon. Utilizing wide-swath altimetry technology, SWOT aims to achieve complete coverage of the world’s oceans and freshwater bodies through high-resolution elevation measurements. SWOT received approval for implementation in 2016, it was ultimately launched in December 2022, and it is currently delivering preliminary data to the public. The primary instrument in SWOT is the Ka-band Radar Interferometer (KaRIn) which utilizes JPL-developed radar interferometry technology to measure ocean and surface water levels with unprecedented accuracy. This paper focuses on the challenges in designing, testing, and finally commissioning in flight a complex instrument like KaRIn. We also present preliminary flight performance and compare it with ground measurements and simulations. Our analysis indicates that KaRIn meets or exceeds all its requirements, but it has also revealed several interesting and unexpected observations, offering just a glimpse of future scientific discoveries that KaRIn will enable.

Modern face recognition systems widely use deep convolutional neural networks (DCNNs). However, DCNNs are susceptible to adversarial examples, posing security risks to these systems. Transferable adversarial examples that can be transferred from surrogate to target models greatly undermine the robustness of DCNNs. Numerous attempts have been made to generate transferable adversarial examples, but the existing methods often suffer from limited transferability or produce adversarial examples with poor image perceptual quality. Recently, diffusion models have shown remarkable success in image generation and have excelled in various downstream tasks. However, their potential in adversarial attacks remains largely unexplored. To bridge this gap, we propose a novel approach, namely Adversarial Diffusion Attack (ADA), in generation of transferable adversarial facial examples. ADA employs a dynamic game-like strategy between injection and denoising that progressively reinforces the robustness of adversarial perturbation in the reverse process of diffusion model. Additionally, both adversarial perturbation and residual image are embedded to drift benign distribution towards adversarial distribution, crafting adversarial examples with high image quality. Extensive experimental results obtained on two benchmarking datasets, LFW and CelebA-HQ, demonstrate that ADA achieves higher attack success rates and produces adversarial examples with superior image quality compared to the state-of-the-art methods.

Performing adversarial attacks on a visual tracker aims to drift the apparent target to the background by adding malicious perturbations to the source images. Demonstrating convincingly their ability to decrease accuracy, existing tracking attackers mislead the target predictions at the decision level, but this is tracker design specific, narrowing their applicability to other tracking approaches. In contrast, we advocate that attacks be performed by corrupting the feature-level clues, i.e., the feature representations extracted by deep networks. The proposed approach provides a general attacking framework for backbone-head tracking architectures. Motivated by the knowledge that the quality of intermediate-level features strongly influences the decision making, four intermediate-level attack methods are proposed to maximise the difference between the feature distributions of natural and adversarial samples, thus decoupling the attack strategies from the form of the output of specific victim trackers. Interestingly, our intermediate-level attacks are compatible with existing decision-level attacks, thus a joint optimisation of these two kinds of adversarial objective functions has the potential to achieve better attacking performance. Hence, the proposed adversarial attack methodology can be used in conjunction with several mainstream tracking paradigms (Discriminative correlation filters, Siamese networks, and Transformer trackers), demonstrating its pluggability. The experimental results on four popular benchmarks, e.g., OTB100, UAV123, LaSOT, and TLP, verify that our method can produce impressive and consistent accuracy degeneration.

Symmetric Positive Definite (SPD) matrices have received wide attention in machine learning due to their intrinsic capacity to encode underlying structural correlation in data. Many successful Riemannian metrics have been proposed to reflect the non-Euclidean geometry of SPD manifolds. However, most existing metric tensors are fixed, which might lead to sub-optimal performance for SPD matrix learning, especially for deep SPD neural networks. To remedy this limitation, we leverage the commonly encountered pullback techniques and propose Adaptive Log-Euclidean Metrics (ALEMs), which extend the widely used Log-Euclidean Metric (LEM). Compared with the previous Riemannian metrics, our metrics contain learnable parameters, which can better adapt to the complex dynamics of Riemannian neural networks with minor extra computations. We also present a complete theoretical analysis to support our ALEMs, including algebraic and Riemannian properties. The experimental and theoretical results demonstrate the merit of the proposed metrics in improving the performance of SPD neural networks. The efficacy of our metrics is further showcased on a set of recently developed Riemannian building blocks, including Riemannian batch normalization, Riemannian Residual blocks, and Riemannian classifiers.

Effectively evaluating the perceptual quality of dehazed images remains an under-explored research issue. In this paper, we propose a no-reference complex-valued convolutional neural network (CV-CNN) model to conduct automatic dehazed image quality evaluation. Specifically, a novel CV-CNN is employed that exploits the advantages of complex-valued representations, achieving better generalization capability on perceptual feature learning than real-valued ones. To learn more discriminative features to analyze the perceptual quality of dehazed images, we design a dual-stream CV-CNN architecture. The dual-stream model comprises a distortion-sensitive stream that operates on the dehazed RGB image, and a haze-aware stream on a novel dark channel difference image. The distortion-sensitive stream accounts for perceptual distortion artifacts, while the haze-aware stream addresses the possible presence of residual haze. Experimental results on three publicly available dehazed image quality assessment (DQA) databases demonstrate the effectiveness and generalization of our proposed CV-CNN DQA model as compared to state-of-the-art no-reference image quality assessment algorithms.

Video action recognition aims at classifying the action category in given videos. In general, semantic-relevant video frame pairs reflect significant action patterns such as object appearance variation and abstract temporal concepts like speed, rhythm, and so on. However, existing action recognition approaches tend to holistically extract spatiotemporal features. Though effective, there is still a risk of neglecting the crucial action features occurring across frames with a long-term temporal span. Motivated by this, in this article, we propose to perceive actions via frame pairs directly and devise a novel Nest Structure with frame pairs as basic units. Specifically, we decompose a video sequence into all possible frame pairs and hierarchically organize them according to temporal frequency and order, thus transforming the original video sequence into a Nest Structure. Through naturally decomposing actions, the proposed structure can flexibly adapt to diverse action variations such as speed or rhythm changes. Next, we devise a Temporal Pair Analysis module (TPA) to extract discriminative action patterns based on the proposed Nest Structure. The designed TPA module consists of a pair calculation part to calculate the pair features and a pair fusion part to hierarchically fuse the pair features for recognizing actions. The proposed TPA can be flexibly integrated into existing backbones, serving as a side branch to capture various action patterns from multi-level features. Extensive experiments show that the proposed TPA module can achieve consistent improvements over several typical backbones, reaching or updating CNN-based SOTA results on several challenging action recognition benchmarks.

Heuristic algorithms have been developed to find approximate solutions for high-utility itemset mining (HUIM) problems that compensate for the performance bottlenecks of exact algorithms. However, heuristic algorithms still face the problem of long runtime and insufficient mining quality, especially for large transaction datasets with thousands to tens of thousands of items and up to millions of transactions. To solve these problems, a novel GPU-based efficient parallel heuristic algorithm for HUIM (PHA-HUIM) is proposed in this paper. The iterative process of PHA-HUIM consists of three main steps: the search strategy, fitness evaluation, and ring topology communication. The search strategy and ring topology communication are designed to run in constant time on GPU. The parallelism of fitness evolution helps to substantially accelerate the algorithm. A new data structure with a sort-mapping strategy is proposed to enhance the search ability and reduce memory usage. To improve the mining quality, a multi-start strategy with an unbalanced allocation strategy is employed in the search process. Ring topology communication is adopted to maintain population diversity. A load balancing strategy is introduced to reduce the thread divergence to improve the parallel efficiency. The experimental results on nine large datasets show that PHA-HUIM outperforms state-of-the-art HUIM algorithms in terms of speedup performance, runtime, and mining quality.

Given a set of multi-view instances, the prevailing assumption in most existing clustering approaches is that they are complete and exhibit cross-view alignment. However, this assumption is often unrealistic. In such scenarios, it could be satisfied at the cost of data pre-processing, but this would be complex and inconsistent with practical applications. Therefore, developing more effective solutions for the View-unaligned Problem (VuP) is highly desirable. Several pioneering works have tackled the partially VuP, yet handling fully VuP remains a challenge due to the reliance on partially pre-aligned instances. In this paper, we propose One-pass View-unaligned Clustering (OpVuC) that simultaneously aligns and clusters instances in a unified framework. Specifically, we alig shuffled instances with a selected template using an innovative global-local alignment scheme based on the notion of geometric invariance and separate the fully aligned instances using a relaxed$k$-means algorithm. The proposed OpVuC method can handle VuP at any alignment level without requiring any pre-aligned instances. Extensive experiments conducted on several benchmark datasets demonstrate the effectiveness and merits of the proposed OpVuC method.

By characterizing each image set as a nonsingular covariance matrix on the symmetric positive definite (SPD) manifold, the approaches of visual content classification with image sets have made impressive progress. However, the key challenge of unhelpfully large intraclass variability and interclass similarity of representations remains open to date. Although, several recent studies have mitigated the two problems by jointly learning the embedding mapping and the similarity metric on the original SPD manifold, their inherent shallow and linear feature transformation mechanism are not powerful enough to capture useful geometric features, especially in complex scenarios. To this end, this article explores a novel approach, termed SPD manifold deep metric learning (SMDML), for image set classification. Specifically, SMDML first selects a prevailing SPD manifold neural network (SPDNet) as the backbone (encoder) to derive an SPD matrix nonlinear representation. To counteract the degradation of structural information during multistage feature embedding, we construct a Riemannian decoder at the end of the encoder, trained by a reconstruction error term (RT), to induce the generated low-dimensional feature manifold of the hidden layer to capture the pivotal information about the visual data describing the imaged scene. We demonstrate through theory and experiments that it is feasible to replace the Riemannian metric with Euclidean distance in RT. Then, the ReCov layer is introduced into the established Riemannian network to regularize the local statistical information within each input feature matrix, which enhances the effectiveness of the learning process. The theoretical analysis of the activation function used in the ReCov layer in terms of continuity and conditions for generating positive definite matrices is beneficial for network design. Inspired by the fact that the single cross-entropy loss used for training is unable to effectively parse the geometric distribution of the deep representations, we finally endow the suggested model with a novel metric learning regularization term. By explicitly incorporating the encoding and processing of the data variations into the network learning process, this term can not only derive a powerful Riemannian representation but also train an effective classifier. The experimental results show the superiority of the proposed approach on three typical visual classification tasks.

Due to the limitations of visible (RGB) sensors in challenging scenarios, such as nighttime and foggy environments, the thermal infrared (TIR) modality draws increasing attention as an auxiliary source for robust tracking systems. Currently, the existing methods extract both the RGB and TIR (RGBT) clues in a similar approach, i.e., utilising RGB-pretrained models with or without finetuning, and then aggregate the multi-modal information through a fusion block embedded in a single level. However, the different imaging principles of RGB and TIR data raise questions about the suitability of RGB-pretrained models for thermal data. In this article, it is argued that the modality gap is overlooked, and an alternative training paradigm is proposed for TIR data to ensure consistency between the training and test data, which is achieved by optimising the TIR feature extractor with only TIR data involved. Furthermore, with the goal of making better use of the enhanced thermal representations, a multi-level fusion strategy is inspired by the observation that various fusion strategies at different levels can contribute to a better performance. Specifically, fusion modules at both the feature and decision levels are derived for a comprehensive fusion procedure while the pixel-level fusion strategy is not considered due to the misalignment of multi-modal image pairs. The effectiveness of our method is demonstrated by extensive qualitative and quantitative experiments conducted on several challenging benchmarks. Code will be released at https://github.com/Zhangyong-Tang/MELT .

On behalf of the organizing committee, we are most delighted to welcome you to join us at 23rd IEEE/ACIS International Conference on Computer and Information Science (ICIS 2023), to be held on June 23-24, 2023 in Wuxi, China. The conference is sponsored by IEEE Computer Society and International Association for Computer and Information Science (ACIS), and in cooperation with Jiangnan University, China.

The Symmetric Positive Definite (SPD) matrices have received wide attention for data representation in many scientific areas. Although there are many different attempts to develop effective deep architectures for data processing on the Riemannian manifold of SPD matrices, very few solutions explicitly mine the local geometrical information in deep SPD feature representations. Given the great success of local mechanisms in Euclidean methods, we argue that it is of utmost importance to ensure the preservation of local geometric information in the SPD networks. We first analyse the convolution operator commonly used for capturing local information in Euclidean deep networks from the perspective of a higher level of abstraction afforded by category theory. Based on this analysis, we define the local information in the SPD manifold and design a multi-scale submanifold block for mining local geometry. Experiments involving multiple visual tasks validate the effectiveness of our approach.

RGB-D object tracking has attracted considerable attention recently, achieving promising performance thanks to the symbiosis between visual and depth channels. However, given a limited amount of annotated RGB-D tracking data, most state-of-the-art RGB-D trackers are simple extensions of high-performance RGB-only trackers, without fully exploiting the underlying potential of the depth channel in the offline training stage. To address the dataset deficiency issue, a new RGB-D dataset named RGBD1K is released in this paper. The RGBD1K contains 1,050 sequences with about 2.5M frames in total. To demonstrate the benefits of training on a larger RGB-D data set in general, and RGBD1K in particular, we develop a transformer-based RGB-D tracker, named SPT, as a baseline for future visual object tracking studies using the new dataset. The results, of extensive experiments using the SPT tracker demonstrate the potential of the RGBD1K dataset to improve the performance of RGB-D tracking, inspiring future developments of effective tracker designs. The dataset and codes will be available on the project homepage: https://github.com/xuefeng-zhu5/RGBD1K.

Most studies on pattern mining have considered only one pattern, such as frequent pattern or high-utility pattern, which is difficult to meet the increasingly diverse needs of users. In this paper, a novel multi-objective problem model for high quality pattern mining (HQPM) is proposed, where the objectives are support, occupancy, and utility. In order to solve the proposed three-objective problem efficiently, an improved multi-objective evolutionary algorithm for HQPM (MOEA-PM) is proposed with two kinds of population initialization strategies and an auxiliary tool. Experimental results on real-world datasets show that the proposed three-objective problem model with the MOEA-PM algorithm can discover patterns that are both frequently occurring and has a high utility in the transaction datasets, while at the same time being relatively complete. The proposed algorithm outperforms the state-of-the-art in terms of efficiency, quality, and convergence speed.

MOTIVATION: Recently, deep learning has become the mainstream methodology for drug-target binding affinity prediction. However, two deficiencies of the existing methods restrict their practical applications. On the one hand, most existing methods ignore the individual information of sequence elements, resulting in poor sequence feature representations. On the other hand, without prior biological knowledge, the prediction of drug-target binding regions based on attention weights of a deep neural network could be difficult to verify, which may bring adverse interference to biological researchers. RESULTS: We propose a novel Multi-Functional and Robust Drug-Target binding Affinity prediction (MFR-DTA) method to address the above issues. Specifically, we design a new biological sequence feature extraction block, namely BioMLP, that assists the model in extracting individual features of sequence elements. Then, we propose a new Elem-feature fusion block to refine the extracted features. After that, we construct a Mix-Decoder block that extracts drug-target interaction information and predicts their binding regions simultaneously. Last, we evaluate MFR-DTA on two benchmarks consistently with the existing methods and propose a new dataset, sc-PDB, to better measure the accuracy of binding region prediction. We also visualize some samples to demonstrate the locations of their binding sites and the predicted multi-scale interaction regions. The proposed method achieves excellent performance on these datasets, demonstrating its merits and superiority over the state-of-the-art methods. AVAILABILITY AND IMPLEMENTATION: https://github.com/JU-HuaY/MFR.

Deep learning based fusion methods have been achieving promising performance in image fusion tasks. This is attributed to the network architecture that plays a very important role in the fusion process. However, in general, it is hard to specify a good fusion architecture, and consequently, the design of fusion networks is still a black art, rather than science. To address this problem, we formulate the fusion task mathematically, and establish a connection between its optimal solution and the network architecture that can implement it. This approach leads to a novel method proposed in the paper of constructing a lightweight fusion network. It avoids the time-consuming empirical network design by a trial-and-test strategy. In particular we adopt a learnable representation approach to the fusion task, in which the construction of the fusion network architecture is guided by the optimisation algorithm producing the learnable model. The low-rank representation (LRR) objective is the foundation of our learnable model. The matrix multiplications, which are at the heart of the solution are transformed into convolutional operations, and the iterative process of optimisation is replaced by a special feed-forward network. Based on this novel network architecture, an end-to-end lightweight fusion network is constructed to fuse infrared and visible light images. Its successful training is facilitated by a detail-to-semantic information loss function proposed to preserve the image details and to enhance the salient features of the source images. Our experiments show that the proposed fusion network exhibits better fusion performance than the state-of-the-art fusion methods on public datasets. Interestingly, our network requires a fewer training parameters than other existing methods.

With the continuously increasing amount of video data, image set classification has recently received widespread attention in the CV&PR community. However, the intra-class diversity and inter-class ambiguity of representations remain an open challenge. To tackle this issue, several methods have been put forward to perform multiple geometry-aware image set modelling and learning. Although the extracted complementary geometric information is beneficial for decision making, the sophisticated computational paradigm (e.g., scatter matrices computation and iterative optimisation) of such algorithms is counterproductive. As a countermeasure, we propose an effective hybrid Riemannian metric learning framework in this paper. Specifically, we design a multiple graph embedding-guided metric learning framework for the sake of fusing these complementary kernel features, obtained via the explicit RKHS embeddings of the Grassmannian manifold, SPD manifold, and Gaussian embedded Riemannian manifold, into a unified subspace for classification. Furthermore, the involved optimisation problem of the developed model can be solved in terms of a series of sub-problems, achieving improved efficiency theoretically and experimentally. Substantial experiments are carried out to evaluate the efficacy of our approach. The experimental results suggest the superiority of it over the state-of-the-art methods.

Recently, deep learning has become the mainstream methodology for Compound-Protein Interaction (CPI) prediction. However, the existing compound-protein feature extraction methods have some issues that limit their performance. First, graph networks are widely used for structural compound feature extraction, but the chemical properties of a compound depend on functional groups rather than graphic structure. Besides, the existing methods lack capabilities in extracting rich and discriminative protein features. Last, the compound-protein features are usually simply combined for CPI prediction, without considering information redundancy and effective feature mining. To address the above issues, we propose a novel CPInformer method. Specifically, we extract heterogeneous compound features, including structural graph features and functional class fingerprints, to reduce prediction errors caused by similar structural compounds. Then, we combine local and global features using dense connections to obtain multi-scale protein features. Last, we apply ProbSparse self-attention to protein features, under the guidance of compound features, to eliminate information redundancy, and to improve the accuracy of CPInformer. More importantly, the proposed method identifies the activated local regions that link a CPI, providing a good visualisation for the CPI state. The results obtained on five benchmarks demonstrate the merits and superiority of CPInformer over the state-of-the-art approaches.

Multi-view subspace clustering is an important topic in cluster analysis. Its aim is to utilize the complementary information conveyed by multiple views of objects to be clustered. Recently, view-shared anchor learning based multi-view clustering methods have been developed to speed up the learning of common data representation. Although widely applied to large-scale scenarios, most of the existing approaches are still faced with two limitations. First, they do not pay sufficient consideration on the negative impact caused by certain noisy views with unclear clustering structures. Second, many of them only focus on the multi-view consistency, yet are incapable of capturing the cross-view diversity. As a result, the learned complementary features may be inaccurate and adversely affect clustering performance. To solve these two challenging issues, we propose a Fast Self-guided Multi-view Subspace Clustering (FSMSC) algorithm which skillfully integrates the view-shared anchor learning and global-guided-local self-guidance learning into a unified model. Such an integration is inspired by the observation that the view with clean clustering structures will play a more crucial role in grouping the clusters when the features of all views are concatenated. Specifically, we first learn a locally-consistent data representation shared by all views in the local learning module, then we learn a globally-discriminative data representation from multi-view concatenated features in the global learning module. Afterwards, a feature selection matrix constrained by the ℓ2,1-norm is designed to construct a guidance from global learning to local learning. In this way, the multi-view consistent and diverse information can be simultaneously utilized and the negative impact caused by noisy views can be overcame to some extent. Extensive experiments on different datasets demonstrate the effectiveness of our proposed fast self-guided learning model, and its promising performance compared to both, the state-of-the-art non-deep and deep multi-view clustering algorithms. The code of this paper is available at https://github.com/chenzhe207/FSMSC.

Advanced Siamese visual object tracking architectures are jointly trained using pair-wise input images to perform target classification and bounding box regression. They have achieved promising results in recent benchmarks and competitions. However, the existing methods suffer from two limitations: First, though the Siamese structure can estimate the target state in an instance frame, provided the target appearance does not deviate too much from the template, the detection of the target in an image cannot be guaranteed in the presence of severe appearance variations. Second, despite the classification and regression tasks sharing the same output from the backbone network, their specific modules and loss functions are invariably designed independently, without promoting any interaction. Yet, in a general tracking task, the centre classification and bounding box regression tasks are collaboratively working to estimate the final target location. To address the above issues, it is essential to perform target-agnostic detection so as to promote cross-task interactions in a Siamese-based tracking framework. In this work, we endow a novel network with a target-agnostic object detection module to complement the direct target inference, and to avoid or minimise the misalignment of the key cues of potential template-instance matches. To unify the multi-task learning formulation, we develop a cross-task interaction module to ensure consistent supervision of the classification and regression branches, improving the synergy of different branches. To eliminate potential inconsistencies that may arise within a multi-task architecture, we assign adaptive labels, rather than fixed hard labels, to supervise the network training more effectively. The experimental results obtained on several benchmarks, i.e., OTB100, UAV123, VOT2018, VOT2019, and LaSOT, demonstrate the effectiveness of the advanced target detection module, as well as the cross-task interaction, exhibiting superior tracking performance as compared with the state-of-the-art tracking methods.

Cross-modal hashing (CMH) has recently received increasing attention with the merit of speed and storage in performing large-scale cross-media similarity search. However, most existing cross-media approaches utilize the batch-based mode to update hash functions, without the ability to efficiently handle the online streaming multimedia data. Online hashing can effectively address the preceding issue by using the online learning scheme to incrementally update the hash functions. Nevertheless, the existing online CMH approaches still suffer from several challenges, such as (1) how to efficiently and effectively utilize the supervision information, (2) how to learn more powerful hash functions, and (3) how to solve the binary constraints. To mitigate these limitations, we present a novel online hashing approach named ONION ( ON line semant I c aut O encoder hashi N g). Specifically, it leverages the semantic autoencoder scheme to establish the correlations between binary codes and labels, delivering the power to obtain more discriminative hash codes. Besides, the proposed ONION directly utilizes the label inner product to build the connection between existing data and newly coming data. Therefore, the optimization is less sensitive to the newly arriving data. Equipping a discrete optimization scheme designed to solve the binary constraints, the quantization errors can be dramatically reduced. Furthermore, the hash functions are learned by the proposed autoencoder strategy, making the hash functions more powerful. Extensive experiments on three large-scale databases demonstrate that the performance of our ONION is superior to several recent competitive online and offline cross-media algorithms.

Bayesian network structure learning (BNSL) from data is an NP-hard problem. Genetic algorithms are powerful for solving combinatorial optimization problems, but the lack of effective guidance results in slow convergence and low accuracy regarding BNSL. To address this problem, we propose a mutual information (MI) guided genetic algorithm (MIGA) for BNSL in this paper, which uses MI to effectively search BN structures. In the initialization phase of MIGA, the population is generated by adding additional constraints based on MI to reach a higher score without losing diversity. By employing normalized MI and defining the population support, the potential dominance in the population can be identified and then used to design a novel crossover operator in order to preserve the dominant genes with a higher probability. Moreover, with the guidance of MI for removing loops from the structures, infeasible solutions can be handled in a straightforward and practical way. The proposed MIGA is evaluated on eleven well-known benchmark datasets and compared with four GA-based methods and four other state-of-the-art BNSL algorithms. Experimental results show that MIGA outperforms the compared algorithms in convergence and learning accuracy.

Due to the explosive growth of multimedia data in recent years, cross-media hashing (CMH) approaches have recently received increasing attention. To learn the hash codes, most existing supervised CMH algorithms employ the strict binary label information, which has small margins between the incorrect labels (0) and the true labels (1), increasing the risk of classification error. Besides, most existing CMH approaches are one-stage algorithms, in which the hash functions and binary codes can be learned simultaneously, complicating the optimization. To avoid NP-hard optimization, many approaches utilize a relaxation strategy. However, this optimisation trick may cause large quantization errors. To address this, we present a novel tWo-stAge discreTe Cross-media Hashing method based on smooth matrix factorization and label relaxation, named WATCH. The proposed WATCH controls the margins adaptively by the novel label relaxation strategy. This innovation reduces the quantization error significantly. Besides, WATCH is a two-stage model. In stage 1, we employ a discrete smooth matrix factorization model. Then, the hash codes can be generated discretely, reducing the large quantization loss greatly. In stage 2, we adopt an effective hash function learning strategy, which produces more effective hash functions. Comprehensive experiments on several datasets demonstrate that WATCH outperforms some state-of-the-art methods.

Given the merits in high computational efficiency and low storage cost, hashing techniques have been widely studied in cross-media retrieval. Existing methods usually adopt the equal length encoding scheme to represent the multimedia data. However, the strictly equal length scheme maybe not optimal because the dimension of different modalities is often various. Besides, there exists other challenges in designing a cross-media retrieval system, e.g., how to address the discrete constraints, how to avoid using the n*n similarity matrix, and how to effectively exploit the discriminative label information. To conquer the above challenges, we propose a novel method, i.e., discrete asymmetric hashing (DAH). Specifically, DAH exploits a flexible model, which can seamlessly deal with equal or unequal length encoding scenarios. Moreover, DAH constructs a supervised semantic embedding framework by jointly minimizing the distance-distance difference and label reconstructing error, significantly reducing the computational complexity. An asymmetric strategy is employed to establish the connection between hash codes and the latent subspace. Furthermore, the hash codes can be learned discretely by the designed optimization algorithm. In the training stage2, a semantic intersection scheme is proposed to learn more powerful hash functions. Experiments show that our DAH is effective in equal and unequal scenarios.

Recently, most dehazed image quality assessment (DQA) methods have focused on estimating remaining haze and omitting distortion impact from the side effect of dehazing algorithms, which leads to their limited performance. Addressing this problem, we propose a method for learning both visibility and distortion-aware features no-reference (NR) dehazed image quality assessment (VDA-DQA). Visibility-aware features are exploited to characterize clarity optimization after dehazing, including the brightness-, contrast-, and sharpness-aware features extracted by the complex contourlet transform (CCT). Then, distortion-aware features are employed to measure the distortion artifacts of images, including the normalized histogram of the local binary pattern (LBP) from the reconstructed dehazed image and the statistics of the CCT subbands corresponding to the chroma and saturation map. Finally, all the above features are mapped into quality scores by support vector regression (SVR). Extensive experimental results on six public DQA datasets verify the superiority of the proposed VDA-DQA method in terms of consistency with subjective visual perception and outperform state-of-the-art methods.

This paper proposes an image fusion framework based on separate representation learning, called IFSepR. We believe that both the co-modal image and the multi-modal image have common and private features based on prior knowledge, exploiting this disentangled representation can help to image fusion, especially to fusion rule design. Inspired by the autoencoder network and contrastive learning, a multi-branch encoder with contrastive constraints is built to learn the common and private features of paired images. In the fusion stage, based on the disentangled features, a general fusion rule is designed to integrate the private features, then combining the fused private features and the common feature are fed into the decoder, reconstructing the fused image. We perform a series of evaluations on three typical image fusion tasks, including multi-focus image fusion, infrared and visible image fusion, medical image fusion. Quantitative and qualitative comparison with five state-of-art image fusion methods demonstrates the advantages of our proposed model.

The dictionary pair learning (DPL) model aims to design a synthesis dictionary and an analysis dictionary to accomplish the goal of rapid sample encoding. In this article, we propose a novel structured representation learning algorithm based on the DPL for image classification. It is referred to as discriminative DPL with scale-constrained structured representation (DPL-SCSR). The proposed DPL-SCSR utilizes the binary label matrix of dictionary atoms to project the representation into the corresponding label space of the training samples. By imposing a non-negative constraint, the learned representation adaptively approximates a block-diagonal structure. This innovative transformation is also capable of controlling the scale of the block-diagonal representation by enforcing the sum of within-class coefficients of each sample to 1, which means that the dictionary atoms of each class compete to represent the samples from the same class. This implies that the requirement of similarity preservation is considered from the perspective of the constraint on the sum of coefficients. More importantly, the DPL-SCSR does not need to design a classifier in the representation space as the label matrix of the dictionary can also be used as an efficient linear classifier. Finally, the DPL-SCSR imposes the$l_{2,p}$-norm on the analysis dictionary to make the process of feature extraction more interpretable. The DPL-SCSR seamlessly incorporates the scale-constrained structured representation learning, within-class similarity preservation of representation, and the linear classifier into one regularization term, which dramatically reduces the complexity of training and parameter tuning. The experimental results on several popular image classification datasets show that our DPL-SCSR can deliver superior performance compared with the state-of-the-art (SOTA) dictionary learning methods. The MATLAB code of this article is available athttps://github.com/chenzhe207/DPL-SCSR.

Nowadays, deep-learning based NLP models are usually trained with large-scale third-party data which can be easily injected with malicious backdoors. Thus, BackDoor Attack (BDA) study has become a trending research to help promote the robustness of an NLP system. Text-based BDA aims to train a poisoned model with both clean and poisoned texts to perform normally on clean inputs while being misled to predict those trigger-embedded texts as target labels set by attackers. Previous works usually choose fixed Positions-to-Poison (P2P) first, then add triggers upon those positions such as letter insertion or deletion. However, considering the positions of words with important semantics may vary in different contexts, fixed P2P models are severely limited in flexibility and performance. We study the text-based BDA from the perspective of automatically and dynamically selecting P2P from contexts. We design a novel Locator model which can predict P2P dynamically without human intervention. Based on the predicted P2P, four effective strategies are introduced to show the BDA performance. Experiments on two public datasets show both tinier test accuracy gap on clean data and higher attack success rate on poisoned ones. Human evaluation with volunteers also shows the P2P predicted by our model are important for classification. Source code is available at https://github.com/jncsnlp/LocatorModel

Abstract With the development of mobile phone camera technology, mobile phones can take a large number of macro photography images that previously could only be taken by professional cameras. Therefore, it is of great significance to study the quality of macro photography images. For this reason, a macro photography image visual quality evaluation database is established and it is named as MP2020. The database contains 100 reference images and 800 distorted images of four distortion types, including 200 distorted images of JPEG 2000, 200 distorted images of JPEG, 200 distorted images of white noise, and 200 distorted images of Gaussian blur. The DMOS values in the database were calculated from 48000 data which are provided by 60 subjects. Ten classical image quality assessment algorithms were tested on the MP2020 database. The experimental results show that the existing image quality assessment algorithms, which are widely used, are not applicable to the macro photography images. Therefore, MP2020 would contribute to the improvement of existing algorithms and the development of new algorithms. MP2020 has been uploaded to GitHub for download.

Abstract A novel deep learning structure of infrared and visible image fusion is proposed. In particular, feature pyramid networks are developed for enhanced feature extraction across multiple convolutional layers. Besides, a fusion strategy is improved based on the channel attention mechanism to highlight the relevant attributes in the fusion stage. The fusion method consists of four parts: encoder, feature pyramid networks, fusion strategy and decoder, respectively. First, the multi‐scale deep features are extracted from the source images by encoder with embedded feature pyramid networks, realizing cross‐layer interaction. Second, these features are fused by the improved fusion strategy with channel attention for each scale. Finally, the fused features are reconstructed by the designed decoder to produce the informative fused image. The experimental results show that the proposed fusion method achieves state‐of‐the‐art results in both qualitative and quantitative evaluation with a lightweight architecture.

Abstract Vulnerabilities can have devastating effects on information security, affecting the economy, social stability, and national security. The idea of automatic vulnerability detection has always attracted researchers. From traditional manual vulnerability mining techniques to static and dynamic detection, all rely on human experts for feature definition. The rapid development of machine learning and deep learning has alleviated the tedious task of manually defining features by human experts while reducing the lack of objectivity caused by human subjective awareness. However, it is still necessary to find an objective characterisation method to define the features of vulnerabilities. Therefore, the authors use code metrics for code characterisation, sequences of metrics representing code. To use code metrics for vulnerability detection, a deep learning‐based vulnerability detection approach that uses a composite neural network of convolutional neural network (CNN) with long short‐term memory (LSTM) is proposed. The authors conduct experiments independently using the proposed approach for CNN‐LSTM CNN, LSTM, gated recurrent units (GRU), and deep neural network (DNN). The authors’ experimental results show that CNN‐LSTM has a high precision of 92%, a recall of 99%, and an accuracy of 91%. In terms of the F1‐score, it is 95%, compared to previous research results, which indicated an improvement of 18%. Compared to other deep learning‐based vulnerability detection models, the authors’ proposed model produced a lower false‐positive rate, a lower miss rate, and improved accuracy.

Vehicular edge computing (VEC) is envisioned as a promising approach to process the explosive computation tasks of vehicular user (VU). In the VEC system, each VU allocates power to process partial tasks through offloading and the remaining tasks through local execution. During the offloading, each VU adopts the multi-input multi-output and non-orthogonal multiple access (MIMO-NOMA) channel to improve the channel spectrum efficiency and capacity. However, the channel condition is uncertain due to the channel interference among VUs caused by the MIMO-NOMA channel and the time-varying path loss caused by the mobility of each VU. In addition, the task arrival of each VU is stochastic in the real world. The stochastic task arrival and uncertain channel condition affect greatly on the power consumption and latency of tasks for each VU. It is critical to design an optimal power allocation scheme considering the stochastic task arrival and channel variation to optimize the long-term reward, including the power consumption and latency in the MIMO-NOMA VEC. Different from the traditional centralized deep reinforcement learning (DRL)-based scheme, this article constructs a decentralized DRL framework to formulate the power allocation optimization problem, where the local observations are selected as the state. The deep deterministic policy gradient (DDPG) algorithm is adopted to learn the optimal power allocation scheme based on the decentralized DRL framework. Simulation results demonstrate that our proposed power allocation scheme outperforms the existing schemes.

Adversarial examples have raised great concerns about the security of deep learning models. Substitute training makes it possible to conduct black-box substitute attacks in real-world scenarios where the attacker does not need access to the structure, parameters, and training set of the target model. However, existing substitute training methods require a large number of queries on the target model and suffer from low attack success rates. To alleviate these problems, we propose a novel black-box adversarial attack method, named substitute meta-learning (SML), which combines meta-learning with the training of the substitute model. Different from existing substitute training methods that rely on data augmentation tactics or refined loss functions, we aim to boost the learning efficiency of the substitute model to improve training efficiency and attack performance. Specifically, we introduce meta-learning to enable the substitute model to learn the knowledge of the target model using a few queries. Extensive experiments are conducted on MNIST and CIFAR-10 datasets. The experimental results show that the proposed SML can improve the attack success rate from 46.1% to 61.3% while requiring fewer queries.

This letter presents a feature alignment method for domain adaptive Acoustic Scene Classification (ASC) across recording devices. First, we design a two-stream network, in which each stream processes two features,i.e., Log-Mel spectrogram and delta-deltas, using two sub-networks. Second, we investigate different loss functions for feature alignment between the feature maps obtained by the source and target domains. Last, we present an alternate training strategy to deal with the data imbalance problem between paired and unpaired samples. The experimental results obtained on the DCASE benchmarks demonstrate the effectiveness and superiority of the proposed method. The source code of the proposed method is available athttps://github.com/Jingqiao-Zhao/FAASC.

The importance of wild video based image set recognition is monotonically increasing due to the large amount of video data being collected by various devices including surveillance cameras, drive recorders, smart phones, and internet. The content of these videos is often complex, and it raises the question of how to perform image set modeling and feature extraction for image set-based classification. In recent years, image set classification methods have advanced considerably by modeling the image set in terms of a covariance matrix, linear subspace, or Gaussian distribution. Moreover, the distinctive geometry spanned by them include Symmetric Positive Definite (SPD) manifold, Grassmannian manifold, and Gaussian embedded Riemannian manifold, respectively. As a matter of fact, most of the approaches just adopt a single geometric model to describe each given image set, which may lose information useful for classification. To tackle this problem, we propose a novel algorithm to model each image set from a multi-geometric perspective. Specifically, the covariance matrix, linear subspace, and Gaussian distribution are applied to set representation simultaneously. In order to fuse these multiple heterogeneous Riemannian manifold-valued features, the well-equipped Riemannian kernel functions are first employed to map them into high dimensional Hilbert spaces. Then, a multi-kernel metric learning framework is devised to embed the learned hybrid kernels into a common lower dimensional subspace to facilitate classification. We conduct experiments on six widely used datasets each representing a different classification task: video-based face recognition, set-based object categorization, video-based emotion recognition, dynamic scene classification, set-based cell identification, and 3D hand pose estimation, to evaluate the classification performance of the proposed algorithm. The extensive experimental results confirm its superiority over the state-of-the-art methods.

In general, flexible ligand docking is used for docking simulations under the premise that the position of the binding site is already known, and meanwhile it can also be used without prior knowledge of the binding site. However, most of the optimization search algorithms used in popular docking software are far from being ideal in the first case, and they can hardly be directly utilized for the latter case due to the relatively large search area. In order to design an algorithm that can flexibly adapt to different sizes of the search area, we propose an effective swarm intelligence optimization algorithm in this paper, called diversity-controlled Lamarckian quantum particle swarm optimization (DCL-QPSO). The highlights of the algorithm are a diversity-controlled strategy and a modified local search method. Integrated with the docking environment of Autodock, the DCL-QPSO is compared with Autodock Vina, Glide and other two Autodock-based search algorithms for flexible ligand docking. Experimental results revealed that the proposed algorithm has a performance comparable to those of Autodock Vina and Glide for dockings within a certain area around the binding sites, and is a more effective solver than all the compared methods for dockings without prior knowledge of the binding sites.

As most human actions in video sequences embody the continuous interactions between foregrounds rather than the background scene, it is significant to disentangle these foregrounds from the background for advanced action recognition systems. In this paper, therefore, we propose a Foreground EXtraction (FEX) block to explicitly model the foreground clues to achieve effective management of action subjects. In particular, the designed FEX block contains two components. The first part is a Foreground Enhancement (FE) module, which highlights the potential feature channels related to the action attributes, providing channel-level refinement for the following spatiotemporal modeling. The second phase is a Scene Segregation (SS) module, which splits feature maps into foreground and background. Specifically, a temporal model with dynamic enhancement is constructed for the foreground part, reflecting the essential nature of the action category. While the background is modeled using simple spatial convolutions, mapping the inputs to the consistent feature space. The FEX blocks can be inserted into existing 2D CNNs (denoted as FEXNet) for spatiotemporal modeling, concentrating on the foreground clues for effective action inference. Our experiments performed on Something-Something V1, V2 and Kinetics400 verify the effectiveness of the proposed method.

Skeleton representation has attracted a great deal of attention recently as an extremely robust feature for human action recognition. However, its non-Euclidean structural characteristics raise new challenges for conventional solutions. Recent studies have shown that there is a native superiority in modeling spatiotemporal skeleton information with a Graph Convolutional Network (GCN). Nevertheless, the skeleton graph modeling normally focuses on the physical adjacency of the elements of the human skeleton sequence, which contrasts with the requirement to provide a perceptually meaningful representation. To address this problem, in this paper, we propose a perceptually-enriched graph learning method by introducing innovative features to spatial and temporal skeleton graph modeling. For the spatial information modeling, we incorporate a Local-Global Graph Convolutional Network (LG-GCN) that builds a multifaceted spatial perceptual representation. This helps to overcome the limitations caused by over-reliance on the spatial adjacency relationships in the skeleton. For temporal modeling, we present a Region-Aware Graph Convolutional Network (RA-GCN), which directly embeds the regional relationships conveyed by a skeleton sequence into a temporal graph model. This innovation mitigates the deficiency of the original skeleton graph models. In addition, we strengthened the ability of the proposed channel modeling methods to extract multi-scale representations. These innovations result in a lightweight graph convolutional model, referred to as Graph2Net, that simultaneously extends the spatial and temporal perceptual fields, and thus enhances the capacity of the graph model to represent skeleton sequences. We conduct extensive experiments on NTU-RGB+D 60&120, Northwestern-UCLA, and Kinetics-400 datasets to show that our results surpass the performance of several mainstream methods while limiting the model complexity and computational overhead.

Hashing methods have sparked great attention on multimedia tasks due to their effectiveness and efficiency. However, most existing methods generate binary codes by relaxing the binary constraints, which may cause large quantization error. In addition, most supervised cross-modal approaches preserve the similarity relationship by constructing an n×n large-size similarity matrix, which requires huge computation, making these methods unscalable. To address the above challenges, this article presents a novel algorithm, called scalable discrete matrix factorization and semantic autoencoder method (SDMSA). SDMSA is a two-stage method. In the first stage, the matrix factorization scheme is utilized to learn the latent semantic information, the label matrix is incorporated into the loss function instead of the similarity matrix. Thereafter, the binary codes can be generated by the latent representations. During optimization, we can avoid manipulating a large n×n similarity matrix, and the hash codes can be generated directly. In the second stage, a novel hash function learning scheme based on the autoencoder is proposed. The encoder-decoder paradigm aims to learn projections, the feature vectors are projected to code vectors by encoder, and the code vectors are projected back to the original feature vectors by the decoder. The encoder-decoder scheme ensures the embedding can well preserve both the semantic and feature information. Specifically, two algorithms SDMSA-lin and SDMSA-ker are developed under the SDMSA framework. Owing to the merit of SDMSA, we can get more semantically meaningful binary hash codes. Extensive experiments on several databases show that SDMSA-lin and SDMSA-ker achieve promising performance.

This article focuses on hyperspectral image (HSI) super-resolution that aims to fuse a low-spatial-resolution HSI and a high-spatial-resolution multispectral image to form a high-spatial-resolution HSI (HR-HSI). Existing deep learning-based approaches are mostly supervised that rely on a large number of labeled training samples, which is unrealistic. The commonly used model-based approaches are unsupervised and flexible but rely on handcrafted priors. Inspired by the specific properties of model, we make the first attempt to design a model-inspired deep network for HSI super-resolution in an unsupervised manner. This approach consists of an implicit autoencoder network built on the target HR-HSI that treats each pixel as an individual sample. The nonnegative matrix factorization (NMF) of the target HR-HSI is integrated into the autoencoder network, where the two NMF parts, spectral and spatial matrices, are treated as decoder parameters and hidden outputs, respectively. In the encoding stage, we present a pixelwise fusion model to estimate hidden outputs directly and then reformulate and unfold the model’s algorithm to form the encoder network. With the specific architecture, the proposed network is similar to a manifold prior-based model and can be trained patch by patch rather than the entire images. Moreover, we propose an additional unsupervised network to estimate the point spread function and spectral response function. Experimental results conducted on both synthetic and real datasets demonstrate the effectiveness of the proposed approach.

Pattern mining (PM) refers to the process of discovering patterns of interest to users from data. However, most studies have considered only one pattern, such as frequent pattern or high-utility pattern. With the continuous requirement of businesses in various industries, the single-objective PM methods are difficult to meet the increasingly diverse needs of users. In this paper, a multi-objective problem model for high quality pattern mining (HQPM) is proposed, where the objectives are support, occupancy, and utility. In order to solve the proposed three-objective problem efficiently, an improved multi-objective evolutionary algorithm for HQPM (MOEA-PM) is proposed. Two kinds of population initialization strategies are designed, which is used to ensure the population is effectively distributed in the feasible solution space. By taking the properties of the model into consideration, an auxiliary tool is proposed to accelerate the convergence of the algorithm. Experimental results on real-world datasets show that the proposed three-objective problem model with the MOEA-PM algorithm can discover patterns that are both frequently occurring and has a high utility in the transaction datasets, while at the same time being relatively complete. Compared with the state-of-the-art MOEA-based HQPM algorithms, MOEA-PM has better performance in terms of efficiency, quality, and convergence speed.

In recent years, attention mechanisms have been widely studied in Discriminative Correlation Filter (DCF) based visual object tracking. To realise spatial attention and discriminative feature mining, existing approaches usually apply regularisation terms to the spatial dimension of multi-channel features. However, these spatial regularisation approaches construct a shared spatial attention pattern for all multi-channel features, without considering the diversity across channels. As each feature map (channel) focuses on a specific visual attribute, a shared spatial attention pattern limits the capability for mining important information from different channels. To address this issue, we advocate channel-specific spatial attention for DCF-based trackers. The key ingredient of the proposed method is an Adaptive Attribute-Aware spatial attention mechanism for constructing a novel DCF-based tracker (A$^3$DCF). To highlight the discriminative elements in each feature map, spatial sparsity is imposed in the filter learning stage, moderated by the prior knowledge regarding the expected concentration of signal energy. In addition, we perform a post processing of the identified spatial patterns to alleviate the impact of less significant channels. The net effect is that the irrelevant and inconsistent channels are removed by the proposed method. The results obtained on a number of well-known benchmarking datasets, including OTB2015, DTB70, UAV123, VOT2018, LaSOT, GOT-10 K and TrackingNet, demonstrate the merits of the proposed A$^3$DCF tracker, with improved performance compared to the state-of-the-art methods.

We propose a novel structured analysis–synthesis dictionary pair learning method for efficient representation and image classification, referred to as relaxed block-diagonal dictionary pair learning with a locality constraint (RBD-DPL). RBD-DPL aims to learn relaxed block-diagonal representations of the input data to enhance the discriminability of both analysis and synthesis dictionaries by dynamically optimizing the block-diagonal components of representation, while the off-block-diagonal counterparts are set to zero. In this way, the learned synthesis subdictionary is allowed to be more flexible in reconstructing the samples from the same class, and the analysis dictionary effectively transforms the original samples into a relaxed coefficient subspace, which is closely associated with the label information. Besides, we incorporate a locality-constraint term as a complement of the relaxation learning to enhance the locality of the analytical encoding so that the learned representation exhibits high intraclass similarity. A linear classifier is trained in the learned relaxed representation space for consistent classification. RBD-DPL is computationally efficient because it avoids both the use of class-specific complementary data matrices to learn discriminative analysis dictionary, as well as the time-consuming$l_{1}/l_{0}$-norm sparse reconstruction process. The experimental results demonstrate that our RBD-DPL achieves at least comparable or better recognition performance than the state-of-the-art algorithms. Moreover, both the training and testing time are significantly reduced, which verifies the efficiency of our method. The MATLAB code of the proposed RBD-DPL is available athttps://github.com/chenzhe207/RBD-DPL.

By representing each image set as a nonsingular covariance matrix on the symmetric positive definite (SPD) manifold, visual classification with image sets has attracted much attention. Despite the success made so far, the issue of large within-class variability of representations still remains a key challenge. Recently, several SPD matrix learning methods have been proposed to assuage this problem by directly constructing an embedding mapping from the original SPD manifold to a lower dimensional one. The advantage of this type of approach is that it cannot only implement discriminative feature selection but also preserve the Riemannian geometrical structure of the original data manifold. Inspired by this fact, we propose a simple SPD manifold deep learning network (SymNet) for image set classification in this article. Specifically, we first design SPD matrix mapping layers to map the input SPD matrices into new ones with lower dimensionality. Then, rectifying layers are devised to activate the input matrices for the purpose of forming a valid SPD manifold, chiefly to inject nonlinearity for SPD matrix learning with two nonlinear functions. Afterward, we introduce pooling layers to further compress the input SPD matrices, and the log-map layer is finally exploited to embed the resulting SPD matrices into the tangent space via log-Euclidean Riemannian computing, such that the Euclidean learning applies. For SymNet, the (2-D)2principal component analysis (PCA) technique is utilized to learn the multistage connection weights without requiring complicated computations, thus making it be built and trained easier. On the tail of SymNet, the kernel discriminant analysis (KDA) algorithm is coupled with the output vectorized feature representations to perform discriminative subspace learning. Extensive experiments and comparisons with state-of-the-art methods on six typical visual classification tasks demonstrate the feasibility and validity of the proposed SymNet.

Recently, hashing-based multimodal learning systems have received increasing attention due to their query efficiency and parsimonious storage costs. However, impeded by the quantization loss caused by numerical optimization, the existing cross-media hashing approaches are unable to capture all the discriminative information present in the original multimodal data. Besides, most cross-modal methods belong to the one-step paradigm, which learn the binary codes and hash function simultaneously, increasing the complexity of optimization. To address these issues, we propose a novel two-stage approach, named the two-stage supervised discrete hashing (TSDH) method. In particular, in the first phase, TSDH generates a latent representation for each modality. These representations are then mapped to a common Hamming space to generate the binary codes. In addition, TSDH directly endows the hash codes with the semantic labels, enhancing the discriminatory power of the learned binary codes. A discrete hash optimization approach is developed to learn the binary codes without relaxation, avoiding the large quantization loss. The proposed hash function learning scheme reuses the semantic information contained by the embeddings, endowing the hash functions with enhanced discriminability. Extensive experiments on several databases demonstrate the effectiveness of the developed TSDH, outperforming several recent competitive cross-media algorithms.

Abstract An aim of completely blind image quality assessment (BIQA) is to develop algorithms which can grade image quality without any prior knowledge of the images. Here, a new contourlet energy statistics based completely on blind opinion‐unaware BIQA (OU‐BIQA) method is proposed, which can predict the perceptual severity of a range of image distortion types without requiring any prior knowledge. According to the energy distribution of the contourlet sub‐bands of natural images in log‐domain, the lower‐scale sub‐band energy can be predicted by the corresponding higher‐scale sub‐band energies of distorted images. A quality model is then constructed by quantifying the difference between predicted energy and realistic energy. Meanwhile, an effective method for adjusting and compensating an undesired distortion is integrated into the quality model. Experimental results show that the proposed new method outperforms state‐of‐the‐art OU‐BIQA models on relevant portions of TID2013 database, and is competitive on the LIVE IQA database. Moreover, the proposed model is very fast, suggesting a real‐time solution to high‐performance BIQA.

Abstract Discriminative Correlation Filters (DCF) have been shown to achieve impressive performance in visual object tracking. However, existing DCF-based trackers rely heavily on learning regularised appearance models from invariant image feature representations. To further improve the performance of DCF in accuracy and provide a parsimonious model from the attribute perspective, we propose to gauge the relevance of multi-channel features for the purpose of channel selection. This is achieved by assessing the information conveyed by the features of each channel as a group, using an adaptive group elastic net inducing independent sparsity and temporal smoothness on the DCF solution. The robustness and stability of the learned appearance model are significantly enhanced by the proposed method as the process of channel selection performs implicit spatial regularisation. We use the augmented Lagrangian method to optimise the discriminative filters efficiently. The experimental results obtained on a number of well-known benchmarking datasets demonstrate the effectiveness and stability of the proposed method. A superior performance over the state-of-the-art trackers is achieved using less than $$10\%$$ 10 % deep feature channels.

We propose an alternating deep-layer cascade (A-DLC) architecture for representation learning in the context of image classification. The merits of the proposed model are threefold. First, A-DLC is the first-ever method that alternatively cascades the sparse and collaborative representations using the class-discriminant softmax vector representation at the interface of each cascade section so that the sparsity and collaborativity can simultaneously be considered. Second, A-DLC inherits the hierarchy learning capability that effectively extends the traditional shallow sparse coding to a multi-layer learning model, thus enabling a full exploitation of the inherent latent discriminative information. Third, the simulation results show a significant amelioration in the classification accuracy, compared to earlier one-step single-layer classification algorithms. The Matlab code of this paper is available at https://github.com/chenzhe207/A-DLC.

In recent years, discriminative correlation filter (DCF) based algorithms have significantly advanced the state of the art in visual object tracking. The key to the success of DCF is an efficient discriminative regression model trained with powerful multi-cue features, including both hand-crafted and deep neural network features. However, the tracking performance is hindered by their inability to respond adequately to abrupt target appearance variations. This issue is posed by the limited representation capability of fixed image features. In this work, we set out to rectify this shortcoming by proposing a complementary representation of a visual content. Specifically, we propose the use of a collaborative representation between successive frames to extract the dynamic appearance information from a target with rapid appearance changes, which results in suppressing the undesirable impact of the background. The resulting collaborative representation coefficients are combined with the original feature maps using a spatially regularised DCF framework for performance boosting. The experimental results on several benchmarking datasets demonstrate the effectiveness and robustness of the proposed method, as compared with a number of state-of-the-art tracking algorithms.

Satellite-derived land surface temperatures (LSTs) are a critical parameter in various fields. Unfortunately, there are numerous gaps in LST products due to cloud contamination and orbital gaps. In previous studies, various gapfilling methods have been developed. However, most of those methods use only spatiotemporal information to fill gaps. In this study, a gapfilling method called the enhanced hybrid (EH) method that integrates spatiotemporal information and information from other similar LST products was proposed. The accuracy of the EH method was compared with the accuracies of three other gapfilling methods that only use spatiotemporal information: Remotely Sensed DAily land Surface Temperature reconstruction (RSDAST), interpolation of the mean anomalies (IMAs), and Gapfill. It was found that the correlations between the four LST products were strong, indicating that using information from other products may improve the accuracy of gapfilling. On average, the mean absolute errors (MAEs) of the data filled using the EH method were 23.7%–52.7% lower than those of RSDAST, 35.4%–38.7% lower than those of IMA, and 38.5%–46.9% lower than those of the Gapfill method. The usage of information from other similar LST products was the main reason for the high accuracy observed for the EH method. In addition, the LST images filled using the RSDAST and IMA methods had some outliers, while there were fewer obvious outliers in the LST images filled with the EH method. It was concluded that the EH method is a robust gapfilling method with a high accuracy.

In the domain of video-based image set classification, a considerable advance has been made by modeling a sequence of video frames (image set) as a linear subspace, which typically resides on a Grassmannian manifold. As a consequence of the large intra-class variations of the video data, there are two open challenges for the modeling task: how to establish appropriate image set models to encode these variations, and how to effectively measure the similarity between any two image sets. As a possible way to tackle these issues, this paper presents a graph embedding multi-kernel metric learning (GEMKML) algorithm for image set classification. The proposed GEMKML implements set modeling, feature extraction, and classification in two steps. Firstly, the proposed framework constructs a novel cascaded feature learning architecture on Grassmannian manifold with the aim of producing more effective Grassmannian manifold-valued feature representations. To make a better use of these learned features, a graph embedding multi-kernel metric learning scheme is then devised to map them into a lower-dimensional Euclidean space, where the inter-class distances are maximized and the intra-class distances are minimized. We evaluate the proposed GEMKML on five different visual classification tasks using widely adopted datasets. The extensive classification results confirm its superiority over the state-of-the-art methods.

This article presents a new Self-growing and Pruning Generative Adversarial Network (SP-GAN) for realistic image generation. In contrast to traditional GAN models, our SP-GAN is able to dynamically adjust the size and architecture of a network in the training stage by using the proposed self-growing and pruning mechanisms. To be more specific, we first train two seed networks as the generator and discriminator; each contains a small number of convolution kernels. Such small-scale networks are much easier and faster to train than large-capacity networks. Second, in the self-growing step, we replicate the convolution kernels of each seed network to augment the scale of the network, followed by fine-tuning the augmented/expanded network. More importantly, to prevent the excessive growth of each seed network in the self-growing stage, we propose a pruning strategy that reduces the redundancy of an augmented network, yielding the optimal scale of the network. Finally, we design a new adaptive loss function that is treated as a variable loss computational process for the training of the proposed SP-GAN model. By design, the hyperparameters of the loss function can dynamically adapt to different training stages. Experimental results obtained on a set of data sets demonstrate the merits of the proposed method, especially in terms of the stability and efficiency of network training. The source code of the proposed SP-GAN method is publicly available at https://github.com/Lambert-chen/SPGAN.git.

Hashing methods have sparked a great revolution on large-scale cross-media search due to its effectiveness and efficiency. Most existing approaches learn unified hash representation in a common Hamming space to represent all multimodal data. However, the unified hash codes may not characterize the cross-modal data discriminatively, because the data may vary greatly due to its different dimensionalities, physical properties, and statistical information. In addition, most existing supervised cross-modal algorithms preserve the similarity relationship by constructing an n × n pairwise similarity matrix, which requires a large amount of calculation and loses the category information. To mitigate these issues, a novel cross-media hashing approach is proposed in this article, dubbed label flexible matrix factorization hashing (LFMH). Specifically, LFMH jointly learns the modality-specific latent subspace with similar semantic by the flexible matrix factorization. In addition, LFMH guides the hash learning by utilizing the semantic labels directly instead of the large n × n pairwise similarity matrix. LFMH transforms the heterogeneous data into modality-specific latent semantic representation. Therefore, we can obtain the hash codes by quantifying the representations, and the learned hash codes are consistent with the supervised labels of multimodal data. Then, we can obtain the similar binary codes of the corresponding modality, and the binary codes can characterize such samples flexibly. Accordingly, the derived hash codes have more discriminative power for single-modal and cross-modal retrieval tasks. Extensive experiments on eight different databases demonstrate that our model outperforms some competitive approaches.

Any high-dimensional data arising from practical applications usually contains irrelevant features that may impact on the performance of existing subspace clustering methods. This paper proposes a novel subspace clustering method which reconstructs the feature matrix by the means of unsupervised feature selection (UFS) to achieve a better dictionary for subspace clustering (SC). Different from most existing clustering methods, the proposed approach uses the reconstructed feature matrix as the dictionary rather than the original data matrix. As the feature matrix reconstructed by representative features is more discriminative and closer to the ground-truth, it results in improved performance. The corresponding non-convex optimization problem is effectively solved using the half-quadratic and augmented Lagrange multiplier methods. Extensive experiments on four real datasets demonstrate the effectiveness of the proposed method.

In recent years, deep learning has been used extensively in the field of image fusion. In this article, we propose a new image fusion method by designing a new structure and a new loss function for a deep learning model. Our backbone network is an autoencoder, in which the encoder has a dual branch structure. We input infrared images and visible light images to the encoder to extract detailed information and semantic information respectively. The fusion layer fuses two sets of features to get fused features. The decoder reconstructs the fusion features to obtain the fused image. We design a new loss function to reconstruct the image effectively. Experiments show that our proposed method achieves state-of-the-art performance.

In recent years, Discriminative Correlation Filters (DCFs) have gained popularity due to their superior performance in visual object tracking. However, existing DCF trackers usually learn filters using fixed attention mechanisms that focus on the centre of an image and suppresses filter amplitudes in surroundings. In this paper, we propose an Adaptive Context-Aware Discriminative Correlation Filter (ACA-DCF) that is able to improve the existing DCF formulation with complementary attention mechanisms. Our ACA-DCF integrates foreground attention and background attention for complementary context-aware filter learning. More importantly, we ameliorate the design using an adaptive weighting strategy that takes complex appearance variations into account. The experimental results obtained on several well-known benchmarks demonstrate the effectiveness and superiority of the proposed method over the state-of-the-art approaches.

In recent years, cross-media retrieval has drawn considerable attention due to the exponential growth of multimedia data. Many hashing approaches have been proposed for the cross-media search task. However, there are still open problems that warrant investigation. For example, most existing supervised hashing approaches employ a binary label matrix, which achieves small margins between wrong labels (0) and true labels (1). This may affect the retrieval performance by generating many false negatives and false positives. In addition, some methods adopt a relaxation scheme to solve the binary constraints, which may cause large quantization errors. There are also some discrete hashing methods that have been presented, but most of them are time-consuming. To conquer these problems, we present a label relaxation and discrete matrix factorization method (LRMF) for cross-modal retrieval. It offers a number of innovations. First of all, the proposed approach employs a novel label relaxation scheme to control the margins adaptively, which has the benefit of reducing the quantization error. Second, by virtue of the proposed discrete matrix factorization method designed to learn the binary codes, large quantization errors caused by relaxation can be avoided. The experimental results obtained on two widely-used databases demonstrate that LRMF outperforms state-of-the-art cross- media methods.

Abstract Efficient and robust facial landmark localisation is crucial for the deployment of real-time face analysis systems. This paper presents a new loss function, namely Rectified Wing (RWing) loss, for regression-based facial landmark localisation with Convolutional Neural Networks (CNNs). We first systemically analyse different loss functions, including L2, L1 and smooth L1. The analysis suggests that the training of a network should pay more attention to small-medium errors. Motivated by this finding, we design a piece-wise loss that amplifies the impact of the samples with small-medium errors. Besides, we rectify the loss function for very small errors to mitigate the impact of inaccuracy of manual annotation. The use of our RWing loss boosts the performance significantly for regression-based CNNs in facial landmarking, especially for lightweight network architectures. To address the problem of under-representation of samples with large pose variations, we propose a simple but effective boosting strategy, referred to as pose-based data balancing. In particular, we deal with the data imbalance problem by duplicating the minority training samples and perturbing them by injecting random image rotation, bounding box translation and other data augmentation strategies. Last, the proposed approach is extended to create a coarse-to-fine framework for robust and efficient landmark localisation. Moreover, the proposed coarse-to-fine framework is able to deal with the small sample size problem effectively. The experimental results obtained on several well-known benchmarking datasets demonstrate the merits of our RWing loss and prove the superiority of the proposed method over the state-of-the-art approaches.

Discriminative correlation filter (DCF) has achieved advanced performance in visual object tracking with remarkable efficiency guaranteed by its implementation in the frequency domain. However, the effect of the structural relationship of DCF and object features has not been adequately explored in the context of the filter design. To remedy this deficiency, this paper proposes a Low-rank and Sparse DCF (LSDCF) that improves the relevance of features used by discriminative filters. To be more specific, we extend the classical DCF paradigm from ridge regression to lasso regression, and constrain the estimate to be of low-rank across frames, thus identifying and retaining the informative filters distributed on a low-dimensional manifold. To this end, specific temporal-spatial-channel configurations are adaptively learned to achieve enhanced discrimination and interpretability. In addition, we analyse the complementary characteristics between hand-crafted features and deep features, and propose a coarse-to-fine heuristic tracking strategy to further improve the performance of our LSDCF. Last, the augmented Lagrange multiplier optimisation method is used to achieve efficient optimisation. The experimental results obtained on a number of well-known benchmarking datasets, including OTB2013, OTB50, OTB100, TC128, UAV123, VOT2016 and VOT2018, demonstrate the effectiveness and robustness of the proposed method, delivering outstanding performance compared to the state-of-the-art trackers.

Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi-level image decomposition method based on latent low-rank representation(LatLRR), which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We build a novel image fusion framework based on MDLatLRR which is used to decompose source images into detail parts(salient features) and base parts. A nuclear-norm based fusion strategy is used to fuse the detail parts and the base parts are fused by an averaging strategy. Compared with other state-of-the-art fusion methods, the proposed algorithm exhibits better fusion performance in both subjective and objective evaluation.

We propose a new Group Feature Selection method for Discriminative Correlation Filters (GFS-DCF) based visual object tracking. The key innovation of the proposed method is to perform group feature selection across both channel and spatial dimensions, thus to pinpoint the structural relevance of multi-channel features to the filtering system. In contrast to the widely used spatial regularisation or feature selection methods, to the best of our knowledge, this is the first time that channel selection has been advocated for DCF-based tracking. We demonstrate that our GFS-DCF method is able to significantly improve the performance of a DCF tracker equipped with deep neural network features. In addition, our GFS-DCF enables joint feature selection and filter learning, achieving enhanced discrimination and interpretability of the learned filters. To further improve the performance, we adaptively integrate historical information by constraining filters to be smooth across temporal frames, using an efficient low-rank approximation. By design, specific temporal-spatial-channel configurations are dynamically learned in the tracking process, highlighting the relevant features, and alleviating the performance degrading impact of less discriminative representations and reducing information redundancy. The experimental results obtained on OTB2013, OTB2015, VOT2017, VOT2018 and TrackingNet demonstrate the merits of our GFS-DCF and its superiority over the state-of-the-art trackers. The code is publicly available at \url{https://github.com/XU-TIANYANG/GFS-DCF}.

Effective data augmentation is crucial for facial landmark localization with convolutional neural networks (CNNs). In this letter, we investigate different data augmentation techniques that can be used to generate sufficient data for training CNN-based facial landmark localization systems. To the best of our knowledge, this is the first study that provides a systematic analysis of different data augmentation techniques in the area. In addition, an online hard augmented example mining (HAEM) strategy is advocated for further performance boosting. We examine the effectiveness of those techniques using a regression-based CNN architecture. The experimental results obtained on the AFLW and COFW datasets demonstrate the importance of data augmentation and the effectiveness of HAEM. The performance achieved using these techniques is superior to the state-of-the-art algorithms.

In this paper, a novel multiobjective evolutionary algorithm (MOEA/CT) is proposed to better manage convergence and distribution of solutions when MOEAs are used for solving multiobjective optimization problems. The coordinate transformation strategy, an external archive update strategy, and a diversity maintenance strategy are proposed in MOEA/CT. The coordinate transformation strategy in the objective space is designed to find more efficient solutions that can accelerate the convergence process. Based on the coordinate transformation strategy, a novel update strategy and diversity maintenance approach for selecting nondominated solutions from the external archive set are integrated in MOEA/CT for getting better distribution of the solutions. The proposed MOEA/CT is compared with eight state-of-art algorithms on six biobjective and seven tri-objective test problems. In terms of four performance metrics, the comparative experimental results demonstrate that MOEA/CT outperforms the other eight competitors and it can achieve solutions with better distribution and better convergence to the Pareto front. In addition, parameter sensitivity analysis is provided to investigate the effect of a key parameter in MOEA/CT; the proposed three strategies are also studied individually to investigate their contribution to MOEA/CT; the performance analysis along with the capacity of external archive is given to clearly make the influence in MOEA/CT; finally, the scalability performance of MOEA/CT is investigated and compared with five notable many-objective evolutionary algorithms on the DTLZ and WFG test suites with 5, 8, 10, and 15 objectives.

In order to effectively identify and classify weld defects of thin-walled metal canisters, a weld defect detection and classification algorithm based on machine vision is proposed in this paper. With the weld defects categorized, a modified background subtraction method based on Gaussian mixture models, is proposed to extract the feature areas of the weld defects. Then, we design an algorithm for weld detection and classification according to the extracted features. Next, by using the weld images sampled by the constructed weld defect detection system on a real-world production line, the parameters of the weld defect classifiers are determined empirically. Experimental results show that the proposed methods can identify and classify the weld defects with more than 95% accuracy rate. Moreover, the weld detection results obtained in the actual production line show that the detection and classification accuracy can reach more than 99%, which means that the system enhanced with the proposed method can meet the requirements for the best real-time and continuous weld defect detection systems available nowadays.

In this paper, we present a novel deep learning architecture for infrared and visible images fusion problem. In contrast to conventional convolutional networks, our encoding network is combined by convolutional layers, fusion layer and dense block in which the output of each layer is connected to every other layer. We attempt to use this architecture to get more useful features from source images in encoding process. And two fusion layers(fusion strategies) are designed to fuse these features. Finally, the fused image is reconstructed by decoder. Compared with existing fusion methods, the proposed fusion method achieves state-of-the-art performance in objective and subjective assessment.

With efficient appearance learning models, discriminative correlation filter (DCF) has been proven to be very successful in recent video object tracking benchmarks and competitions. However, the existing DCF paradigm suffers from two major issues, i.e., spatial boundary effect and temporal filter degradation. To mitigate these challenges, we propose a new DCF-based tracking method. The key innovations of the proposed method include adaptive spatial feature selection and temporal consistent constraints, with which the new tracker enables joint spatial-temporal filter learning in a lower dimensional discriminative manifold. More specifically, we apply structured spatial sparsity constraints to multi-channel filters. Consequently, the process of learning spatial filters can be approximated by the lasso regularization. To encourage temporal consistency, the filter model is restricted to lie around its historical value and updated locally to preserve the global structure in the manifold. Last, a unified optimization framework is proposed to jointly select temporal consistency preserving spatial features and learn discriminative filters with the augmented Lagrangian method. Qualitative and quantitative evaluations have been conducted on a number of well-known benchmarking datasets such as OTB2013, OTB50, OTB100, Temple-Colour, UAV123, and VOT2018. The experimental results demonstrate the superiority of the proposed method over the state-of-the-art approaches.

We present a new loss function, namely Wing loss, for robust facial landmark localisation with Convolutional Neural Networks (CNNs). We first compare and analyse different loss functions including L2, L1 and smooth L1. The analysis of these loss functions suggests that, for the training of a CNN-based localisation model, more attention should be paid to small and medium range errors. To this end, we design a piece-wise loss function. The new loss amplifies the impact of errors from the interval (-w, w) by switching from L1 loss to a modified logarithm function. To address the problem of under-representation of samples with large out-of-plane head rotations in the training set, we propose a simple but effective boosting strategy, referred to as pose-based data balancing. In particular, we deal with the data imbalance problem by duplicating the minority training samples and perturbing them by injecting random image rotation, bounding box translation and other data augmentation approaches. Last, the proposed approach is extended to create a two-stage framework for robust facial landmark localisation. The experimental results obtained on AFLW and 300W demonstrate the merits of the Wing loss function, and prove the superiority of the proposed method over the state-of-the-art approaches.

This paper investigates the evaluation of dense 3D face reconstruction from a single 2D image in the wild. To this end, we organise a competition that provides a new benchmark dataset that contains 2000 2D facial images of 135 subjects as well as their 3D ground truth face scans. In contrast to previous competitions or challenges, the aim of this new benchmark dataset is to evaluate the accuracy of a 3D dense face reconstruction algorithm using real, accurate and high-resolution 3D ground truth face scans. In addition to the dataset, we provide a standard protocol as well as a Python script for the evaluation. Last, we report the results obtained by three state-of-the-art 3D face reconstruction systems on the new benchmark dataset. The competition is organised along with the 2018 13th IEEE Conference on Automatic Face & Gesture Recognition.

In the domain of pattern recognition, using the CovDs (Covariance Descriptors) to represent data and taking the metrics of the resulting Riemannian manifold into account have been widely adopted for the task of image set classification. Recently, it has been proven that infinite-dimensional CovDs are more discriminative than their low-dimensional counterparts. However, the form of infinite-dimensional CovDs is implicit and the computational load is high. We propose a novel framework for representing image sets by approximating infinite-dimensional CovDs in the paradigm of the Nyström method based on a Riemannian kernel. We start by modeling the images via CovDs, which lie on the Riemannian manifold spanned by SPD (Symmetric Positive Definite) matrices. We then extend the Nyström method to the SPD manifold and obtain the approximations of CovDs in RKHS (Reproducing Kernel Hilbert Space). Finally, we approximate infinite-dimensional CovDs via these approximations. Empirically, we apply our framework to the task of image set classification. The experimental results obtained on three benchmark datasets show that our proposed approximate infinite-dimensional CovDs outperform the original CovDs.

In recent years, deep learning has become a very active research tool which is used in many image processing fields. In this paper, we propose an effective image fusion method using a deep learning framework to generate a single image which contains all the features from infrared and visible images. First, the source images are decomposed into base parts and detail content. Then the base parts are fused by weighted-averaging. For the detail content, we use a deep learning network to extract multi-layer features. Using these features, we use$l_{1}$-norm and weighted-average strategy to generate several candidates of the fused detail content. Once we get these candidates, the max selection strategy is used to get the final fused detail content. Finally, the fused image will be reconstructed by combining the fused base part and the detail content. The experimental results demonstrate that our proposed method achieves state-of-the-art performance in both objective assessment and visual quality. The Code of our fusion method is available at https://github.com/exceptionLi/imagefusion_deeplearning.

In image set classification, a considerable advance has been made by modeling the original image sets by second order statistics or linear subspace, which typically lie on the Riemannian manifold. Specifically, they are Symmetric Positive Definite (SPD) manifold and Grassmann manifold respectively, and some algorithms have been developed on them for classification tasks. Motivated by the inability of existing methods to extract discriminatory features for data on Riemannian manifolds, we propose a novel algorithm which combines multiple manifolds as the features of the original image sets. In order to fuse these manifolds, the well-studied Riemannian kernels have been utilized to map the original Riemannian spaces into high dimensional Hilbert spaces. A metric Learning method has been devised to embed these kernel spaces into a lower dimensional common subspace for classification. The state-of-the-art results achieved on three datasets corresponding to two different classification tasks, namely face recognition and object categorization, demonstrate the effectiveness of the proposed method.

Discriminative correlation filter (DCF) based tracking methods have achieved great success recently. However, the temporal learning scheme in the current paradigm is of a linear recursion form determined by a fixed learning rate which can not adaptively feedback appearance variations. In this paper, we propose a unified non-negative subspace representation constrained leaning scheme for DCF. The subspace is constructed by several templates with auxiliary memory mechanisms. Then the current template is projected onto the subspace to find the non-negative representation and to determine the corresponding template weights. Our learning scheme enables efficient combination of correlation filter and subspace structure. The experimental results on OTB50 demonstrate the effectiveness of our learning formulation.

Recently, hashing techniques have gained importance in large-scale retrieval tasks because of their retrieval speed. Most of the existing cross-view frameworks assume that data are well paired. However, the fully-paired multiview situation is not universal in real applications. The aim of the method proposed in this paper is to learn the hashing function for semi-paired cross-view retrieval tasks. To utilize the label information of partial data, we propose a semi-supervised hashing learning framework which jointly performs feature extraction and classifier learning. The experimental results on two datasets show that our method outperforms several state-of-the-art methods in terms of retrieval accuracy.

In pattern recognition, the task of image set classification has often been performed by representing data using symmetric positive definite (SPD) matrices, in conjunction with the metric of the resulting Riemannian manifold. In this paper, we propose a new data representation framework for image sets which we call component symmetric positive definite representation (CSPD). Firstly, we obtain sub-image sets by dividing the images in the set into square blocks of the same size, and use a traditional SPD model to describe them. Then, we use the Riemannian kernel to determine similarities of corresponding subimage sets. Finally, the CSPD matrix appears in the form of the kernel matrix for all the sub-image sets; its i, j-th entry measures the similarity between the i-th and j-th sub-image sets. The Riemannian kernel is shown to satisfy Mercer’s theorem, so the CSPD matrix is symmetric and positive definite, and also lies on a Riemannian manifold. Test on three benchmark datasets shows that CSPD is both lower-dimensional and more discriminative data descriptor than standard SPD for the task of image set classification.

The IEEE Smart City Initiative was created in 2013 to develop a worldwide network of cities, sharing their experience and organizing knowledge dissemination in their respective ecosystems. After more than three years, almost 15 cities joined the five core cities, creating a vibrant network of people, keen on producing and sharing knowledge, organizing events, and spreading the smart cities' culture around the world.

3D Morphable Face Models (3DMM) have been used in pattern recognition for some time now. They have been applied as a basis for 3D face recognition, as well as in an assistive role for 2D face recognition to perform geometric and photometric normalisation of the input image, or in 2D face recognition system training. The statistical distribution underlying 3DMM is Gaussian. However, the single-Gaussian model seems at odds with reality when we consider different cohorts of data, e.g. Black and Chinese faces. Their means are clearly different. This paper introduces the Gaussian Mixture 3DMM (GM-3DMM) which models the global population as a mixture of Gaussian subpopulations, each with its own mean. The proposed GM-3DMM extends the traditional 3DMM naturally, by adopting a shared covariance structure to mitigate small sample estimation problems associated with data in high dimensional spaces. We construct a GM-3DMM, the training of which involves a multiple cohort dataset, SURREY-JNU, comprising 942 3D face scans of people with mixed backgrounds. Experiments in fitting the GM-3DMM to 2D face images to facilitate their geometric and photometric normalisation for pose and illumination invariant face recognition demonstrate the merits of the proposed mixture of Gaussians 3D face model.

The paper presents a dictionary integration algorithm using 3D morphable face models (3DMM) for pose-invariant collaborative-representation-based face classification. To this end, we first fit a 3DMM to the 2D face images of a dictionary to reconstruct the 3D shape and texture of each image. The 3D faces are used to render a number of virtual 2D face images with arbitrary pose variations to augment the training data, by merging the original and rendered virtual samples to create an extended dictionary. Second, to reduce the information redundancy of the extended dictionary and improve the sparsity of reconstruction coefficient vectors using collaborative-representation-based classification (CRC), we exploit an on-line class elimination scheme to optimise the extended dictionary by identifying the training samples of the most representative classes for a given query. The final goal is to perform pose-invariant face classification using the proposed dictionary integration method and the on-line pruning strategy under the CRC framework. Experimental results obtained for a set of well-known face data sets demonstrate the merits of the proposed method, especially its robustness to pose variations.

Large scale optimization has become a well-recognised field in many science and engineering applications and a variety of metaheuristic algorithms adopting cooperative coevolution (CC) framework with problem decomposition have been applied to solve them. In this paper, a novel decomposition strategy termed as random selection is proposed. In random selection strategy, only a small part of decision variables are randomly selected to form a group for evolving at every iteration and the maximum number of randomly selected decision variables are limited by the parameter RSSCALE. By random selection, the randomly selected searching subspace is explored sufficiently in each iteration and the whole search space can be fully covered after several iterations. We evaluate the random selection strategy by combining quantum-behaved particle swarm optimization (RSQPSO) and a comparative study is carried out on a set of benchmark functions between RSQPSO and four state-of-the-art algorithms, which were specially designed for large scale optimization. The comparative results show that the proposed approach performs well for solving large scale optimization problems.

We present a new Cascaded Shape Regression (CSR) architecture, namely Dynamic Attention-Controlled CSR (DAC-CSR), for robust facial landmark detection on unconstrained faces. Our DAC-CSR divides facial landmark detection into three cascaded sub-tasks: face bounding box refinement, general CSR and attention-controlled CSR. The first two stages refine initial face bounding boxes and output intermediate facial landmarks. Then, an online dynamic model selection method is used to choose appropriate domain-specific CSRs for further landmark refinement. The key innovation of our DAC-CSR is the fault-tolerant mechanism, using fuzzy set sample weighting, for attention-controlled domain-specific model training. Moreover, we advocate data augmentation with a simple but effective 2D profile face generator, and context-aware feature extraction for better facial feature representation. Experimental results obtained on challenging datasets demonstrate the merits of our DAC-CSR over the state-of-the-art methods.

Recently, low‐rank representation (LRR)‐based techniques have manifested remarkable results for data representation. To exploit the latent manifold structure of data, the graph regulariser is incorporated into the model of LRR. However, it is critical to construct an appropriate graph model and set the corresponding parameters. In addition, this procedure is usually time‐consuming and proved to be overfitting when using cross validation or discrete grid search. Two novel LRR‐based methods, called multiple graph regularised LRR and multiple hypergraph regularised LLR, are proposed to represent the high‐dimensional data. To guarantee the smoothness along the estimated manifold, the multiple graph regulariser and the multiple hypergraph regulariser are incorporated into the traditional LRR method, respectively, which results in a unified framework. Moreover, the augmented Lagrange multiplier is adopted to solve the proposed models. Extensive experiments on real image datasets show the effectiveness of the proposed methods.

This paper presents a half-face dictionary integration (HFDI) algorithm for representation-based classification. The proposed HFDI algorithm measures residuals between an input signal and the reconstructed one, using both the original and the synthesized dual-column (row) half-face training samples. More specifically, we first generate a set of virtual half-face samples for the purpose of training data augmentation. The aim is to obtain high-fidelity collaborative representation of a test sample. In this half-face integrated dictionary, each original training vector is replaced by an integrated dual-column (row) half-face matrix. Second, to reduce the redundancy between the original dictionary and the extended half-face dictionary, we propose an elimination strategy to gain the most robust training atoms. The last contribution of the proposed HFDI method is the use of a competitive fusion method weighting the reconstruction residuals from different dictionaries for robust face classification. Experimental results obtained from the Facial Recognition Technology, Aleix and Robert, Georgia Tech, ORL, and Carnegie Mellon University-pose, illumination and expression data sets demonstrate the effectiveness of the proposed method, especially in the case of the small sample size problem.

An opposition-based learning competitive particle swarm optimizer (OBL-CPSO) is proposed to address the problem of premature convergence in PSO. Two learning mechanisms have been employed in OBL-CPSO, which are competitive learning from competitive swarm optimizer (CSO) and opposition-based learning. In each iteration of OBL-CPSO, the competitive learning works among three randomly selected particles from the swarm and is followed by the comparison of fitness. The particle with the best fitness, denoted as the winner, is passed to the next iteration directly. The particle with the worst fitness learns from the winner and the particle with medium fitness takes the opposition-based learning for quickly exploiting the search space and then they are passed to the next iteration with updated positions and velocities. The performance of OBL-CPSO is evaluated on a set of benchmark functions and compared with PSO, six traditional PSO variants, and four state-of-art PSO variants. Experimental results show that OBL-CPSO has the best performance on the quality of solution and the convergence speed among the compared algorithms. The OBL-CPSO also has scalability to large scale global optimization problems from the experimental results on problems with 100, 300, and 500 dimensions. The OBL-CPSO is proposed to solve the learning problem of fuzzy cognitive maps (FCMs) which is a kind of complex networks and shows advantage over the compared algorithms for the problem.

To avoid the introduction of false information during the fusion progress, a novel multi-focus image fusion method is proposed in quaternion wavelet transform domain. To obtain the dependency in different high frequency subbands, a quaternion wavelet contextual hidden Markov model (Q-CHMM) is established for modeling quaternion wavelet coefficients. And for better image representations, several features are proposed by analyzing the transform coefficients, phases of coefficients and the statistical attribution of coefficients. Different from the traditional fusion methods basing on a single feature, a comprehensive feature is constructed by using quaternion matrix to fuse the high frequency subbands. Experimental results demonstrate that the proposed method possess good fusion performance.

Low-rank representation (LRR) and its variations have recently attracted a great deal of attention because of its effectiveness in exploring low-dimensional subspace structures embedded in data. LRR-related algorithms have many applications in computer vision, signal processing, semi-supervised learning and pattern recognition. However, most of the existing LRR methods fail to take into account the non-linear geometric structures within data, thus the locality and the similarity information among data may be missing in the learning process, which have been shown to be beneficial for discriminative tasks. To improve LRR in this regard, we propose a manifold locality constrained low-rank representation framework (MLCLRR) for data representation. By taking the local manifold structure of the data into consideration, the proposed MLCLRR method not only can represent the global low-dimensional structures, but also capture the local intrinsic non-linear geometric information in the data. The experimental results on different types of vision problems demonstrate the effectiveness of the proposed method.

We propose a new learning quality-aware features (LQAF) blind image quality assessment (IQA) algorithm for multiply distorted images. In the new model, some relevant quality perceptual features, including mean value of intensity, contrast sensitivity function (CSF), mean value of gradient magnitude, and 15 texture parameters of gray level-gradient co-occurrence matrix (GGCM) from four categories images: original distorted image and its phase congruency (PC) image, covariance maximum and minimum image of phase congruency, are used. Image quality estimation is accomplished by the approximating function between these features and subjective mean opinion scores using support vector regression (SVR). Experimental results on the LIVE multiply distorted image database (LIVEMD) demonstrate the effectiveness of our proposed method.

In this letter, we present a random cascaded-regression copse (R-CR-C) for robust facial landmark detection. Its key innovations include a new parallel cascade structure design, and an adaptive scheme for scale-invariant shape update and local feature extraction. Evaluation on two challenging benchmarks shows the superiority of the proposed algorithm to state-of-the-art methods.

A large amount of training data is usually crucial for successful supervised learning. However, the task of providing training samples is often time-consuming, involving a considerable amount of tedious manual work. In addition, the amount of training data available is often limited. As an alternative, in this paper, we discuss how best to augment the available data for the application of automatic facial landmark detection. We propose the use of a 3D morphable face model to generate synthesized faces for a regression-based detector training. Benefiting from the large synthetic training data, the learned detector is shown to exhibit a better capability to detect the landmarks of a face with pose variations. Furthermore, the synthesized training data set provides accurate and consistent landmarks automatically as compared to the landmarks annotated manually, especially for occluded facial parts. The synthetic data and real data are from different domains; hence the detector trained using only synthesized faces does not generalize well to real faces. To deal with this problem, we propose a cascaded collaborative regression algorithm, which generates a cascaded shape updater that has the ability to overcome the difficulties caused by pose variations, as well as achieving better accuracy when applied to real faces. The training is based on a mix of synthetic and real image data with the mixing controlled by a dynamic mixture weighting schedule. Initially, the training uses heavily the synthetic data, as this can model the gross variations between the various poses. As the training proceeds, progressively more of the natural images are incorporated, as these can model finer detail. To improve the performance of the proposed algorithm further, we designed a dynamic multi-scale local feature extraction method, which captures more informative local features for detector training. An extensive evaluation on both controlled and uncontrolled face data sets demonstrates the merit of the proposed algorithm.

In this paper, a novel region segmentation and sigmoid function based image fusion method is proposed. Different from the traditional fusion approaches limiting to a single fusion strategy, the proposed method is designed with an adaptive multi-strategy fusion rule (AMFR). In our method, the source images are decomposed into low frequency sub bands and high frequency sub bands via the shift-invariant Shear let transform (SIST). The low frequency sub bands are fused by the choose-max scheme and the high frequency sub bands are fused by the AMFR based on a sigmoid function. The AMFR includes the choose-max scheme and the weighted average scheme, which of them is selected is determined by the sigmoid function. The fused sub bands are merged to reconstruct fused image by using inverse SIST. Experiments conducted on various types of source images demonstrate that our approach achieve superior results compared with the existing fusion methods in both visual presentation and objective evaluation.

In this paper, a new Contextual hidden Markov Model (CHMM) and modified Pulse Coupled Neural Network (M-PCNN) based fusion approach in the Contour domain is proposed for multi-modal medical image fusion. The Contour transform as an emerging multi-scale multi-direction geometric analyzing tool can provide an efficient and flexible representation of images, e.g. edges, contours and textures, which overcomes the drawback of the 2-D wavelet transform. Considering the powerful advantages for statistical modeling and processing of Contour let coefficients by HMM, the context information integrated with HMM is established to construct a comprehensive statistical correlative model, which can collectively capture persistence across scales, directional selectivity within scales and energy concentration in the spatial neighborhood of the high-frequency sub-band coefficients. Low-frequency sub-band coefficients are fused by the magnitude maximum rule, and a modified PCNN is developed where the linking strength of each neuron is determined by the normalized region energy of Edge PDF and modified spatial frequency is employed as the image feature to motivate M-PCNN. The high-frequency directional sub-band coefficients are selected by total pulse number maximum strategy. The experimental results demonstrate that the presented fusion method can further improve fusion image quality and visual effects.

BACKGROUND: Finding an efficient method to solve the parameter estimation problem (inverse problem) for nonlinear biochemical dynamical systems could help promote the functional understanding at the system level for signalling pathways. The problem is stated as a data-driven nonlinear regression problem, which is converted into a nonlinear programming problem with many nonlinear differential and algebraic constraints. Due to the typical ill conditioning and multimodality nature of the problem, it is in general difficult for gradient-based local optimization methods to obtain satisfactory solutions. To surmount this limitation, many stochastic optimization methods have been employed to find the global solution of the problem. RESULTS: This paper presents an effective search strategy for a particle swarm optimization (PSO) algorithm that enhances the ability of the algorithm for estimating the parameters of complex dynamic biochemical pathways. The proposed algorithm is a new variant of random drift particle swarm optimization (RDPSO), which is used to solve the above mentioned inverse problem and compared with other well known stochastic optimization methods. Two case studies on estimating the parameters of two nonlinear biochemical dynamic models have been taken as benchmarks, under both the noise-free and noisy simulation data scenarios. CONCLUSIONS: The experimental results show that the novel variant of RDPSO algorithm is able to successfully solve the problem and obtain solutions of better quality than other global optimization methods used for finding the solution to the inverse problems in this study.

Hidden Markov Models (HMMs) are powerful tools for multiple sequence alignment (MSA), which is known to be an NP-complete and important problem in bioinformatics. Learning HMMs is a difficult task, and many meta-heuristic methods, including particle swarm optimization (PSO), have been used for that. In this paper, a new variant of PSO, called the random drift particle swarm optimization (RDPSO) algorithm, is proposed to be used for HMM learning tasks in MSA problems. The proposed RDPSO algorithm, inspired by the free electron model in metal conductors in an external electric field, employs a novel set of evolution equations that can enhance the global search ability of the algorithm. Moreover, in order to further enhance the algorithmic performance of the RDPSO, we incorporate a diversity control method into the algorithm and, thus, propose an RDPSO with diversity-guided search (RDPSO-DGS). The performances of the RDPSO, RDPSO-DGS and other algorithms are tested and compared by learning HMMs for MSA on two well-known benchmark data sets. The experimental results show that the HMMs learned by the RDPSO and RDPSO-DGS are able to generate better alignments for the benchmark data sets than other most commonly used HMM learning methods, such as the Baum-Welch and other PSO algorithms. The performance comparison with well-known MSA programs, such as ClustalW and MAFFT, also shows that the proposed methods have advantages in multiple sequence alignment.

This paper proposes the random drift particle swarm optimization (RDPSO) algorithm to solve economic dispatch (ED) problems from power systems area. The RDPSO is inspired by the free electron model in metal conductors placed in an external electric field, and it employs a novel set of evolution equations that can enhance the global search ability of the algorithm. Many nonlinear characteristics of a power generator, such as the ramp rate limits, prohibited operating zones and nonsmooth cost functions are considered when the proposed method is used in practice for optimizing the generators' operation. The performance of the RDPSO method is evaluated on three different power systems, and compared with that of other optimization methods in terms of the solution quality, robustness, and convergence performance. The experimental results show that the RDPSO method performs better in solving the ED problems than any other tested optimization techniques.

This paper introduces a variant of particle swarm optimization algorithm called the drift particle swarm optimization (DPSO), which is inspired by the free electron model in an external electric field at finite temperature. As the compression-expansion coefficient in DPSO is an important parameter which can greatly influence the performance of the algorithm, three types of control strategies are proposed to control this parameter. The performance of these strategies on the DPSO is comprehensively evaluated on eight benchmark functions. From the experimental results and statistical tests, guidelines about selecting the control method for the compression-expansion coefficient are given.

Quantum-behaved particle swarm optimization (QPSO), motivated by concepts from quantum mechanics and particle swarm optimization (PSO), is a probabilistic optimization algorithm belonging to the bare-bones PSO family. Although it has been shown to perform well in finding the optimal solutions for many optimization problems, there has so far been little analysis on how it works in detail. This paper presents a comprehensive analysis of the QPSO algorithm. In the theoretical analysis, we analyze the behavior of a single particle in QPSO in terms of probability measure. Since the particle's behavior is influenced by the contraction-expansion (CE) coefficient, which is the most important parameter of the algorithm, the goal of the theoretical analysis is to find out the upper bound of the CE coefficient, within which the value of the CE coefficient selected can guarantee the convergence or boundedness of the particle's position. In the experimental analysis, the theoretical results are first validated by stochastic simulations for the particle's behavior. Then, based on the derived upper bound of the CE coefficient, we perform empirical studies on a suite of well-known benchmark functions to show how to control and select the value of the CE coefficient, in order to obtain generally good algorithmic performance in real world applications. Finally, a further performance comparison between QPSO and other variants of PSO on the benchmarks is made to show the efficiency of the QPSO algorithm with the proposed parameter control and selection methods.

We develop a no-reference image quality assessment (QA) algorithm that deploys a general regression neural network (GRNN). The new algorithm is trained on and successfully assesses image quality, relative to human subjectivity, across a range of distortion types. The features deployed for QA include the mean value of phase congruency image, the entropy of phase congruency image, the entropy of the distorted image, and the gradient of the distorted image. Image quality estimation is accomplished by approximating the functional relationship between these features and subjective mean opinion scores using a GRNN. Our experimental results show that the new method accords closely with human subjective judgment.

We propose a new kernel direct discriminant analysis (KDDA) algorithm in this paper. First, a recently advocated direct linear discriminant analysis (DLDA) algorithm is overviewed. Then the new KDDA algorithm is developed which can be considered as a kernel version of the DLDA algorithm. The design of the minimum distance classifier in the new kernel subspace is then discussed. The results of experiments on two well-known facial databases show the effectiveness of the proposed method in face recognition. The results of experiments also confirm that DLDA can be viewed as a special case of the proposed KDDA algorithm. 1.