The substantial heterogeneity in energy and communication protocols among IoT devices leads to highly diversified image resolutions, which severely constrain the reliability of downstream visual analysis tasks in applications like intelligent transportation and smart buildings. While recent deep learning-based SR methods have achieved remarkable success, the majority are typically designed for specific integer scaling factors, requiring separate models for different scales, which is impractical for real-world IoT applications. To address this, we propose SafaSR, an arbitrary-scale image super-resolution network based on multi-domain feature fusion. Our key innovation lies in a Multi-domain Multi-level Feature Fusion(M2F2 )mechanism, which is driven by a scale-aware feature learning (SFL) model that adaptively extracts features from both spatial and frequency domains. TheM2F2mechanism is designed to reduce correlations between different feature domains, allowing a more effective integration of complementary information. Extensive experiments show that our proposed network outperforms the most advanced image SR algorithms in terms of PSNR and SSIM metrics on the benchmark datasets, with fewer network parameters and less runtime.

Functional brain connectivity networks capture complex relationships and temporal evolution between brain regions, which have become increasingly important for diagnosing neurological disorders. However, existing methods, which are primarily based on vector or graph representations, struggle to adequately characterize the intricate spatio-temporal topological architecture of functional brain networks. Additionally, they predominantly rely on data-driven paradigms and lack priors pertaining to cross-windows network interactions. To address these issues, we propose a spatio-temporal hypergraph attention network framework for brain network analysis. Specifically, we first propose a temporal attention network architecture embedded with temporal similarity-driven prior knowledge, which effectively extracts long-range dependency information from fMRI by combining multi-head self-attention mechanisms and cross-window temporal prior knowledge. Second, we design a hierarchical hypergraph generation module that fuses local and global brain topological information to achieve multi-scale modeling of high-order spatio-temporal structures. Additionally, the spatial attention network, developed based on transformer architecture, employs hypergraph message passing mechanisms to effectively construct multi-level spatial interaction relationships between brain regions. Finally, a multi-layer perceptron (MLP) is adopted for classification. Experiments on the ADNI and PD datasets demonstrate that our method outperforms several state-of-the-art approaches in diagnostic performance and provides discriminative graph features for relevant brain disease diagnosis.

Deep neural networks enriched with structural information have been widely employed for facial expression recognition tasks. However, these methods often depend on hierarchical information rather than face property to finish expression recognition. In this paper, we propose a cross-modal network with strong biological and structural information for facial expression recognition (CMNet). CMNet can respectively learn expression information via face symmetry on a whole face, left and right half faces to extract complementary facial features. To prevent negative effect of biological and structural information fusion, a salient facial information refinement module can obtain salient facial expression information to improve stability of an obtained facial expression classifier. To reduce reliance on unilateral facial features, a half-face alignment optimization mechanism is designed to align obtained expression information of learned left and right half faces. Our experimental results demonstrate that CMNet outperforms several novel methods, i.e., SCN and LAENet-SA for facial expression recognition. Codes can be obtained at https://github.com/hellloxiaotian/CMNet.

Deep convolutional neural networks can use hierarchical information to progressively extract structural information to recover high-quality images. However, preserving the effectiveness of the obtained structural information is important in image super-resolution. In this paper, we propose a cosine network for image super-resolution (CSRNet) by improving a network architecture and optimizing the training strategy. To extract complementary homologous structural information, odd and even heterogeneous blocks are designed to enlarge the architectural differences and improve the performance of image super-resolution. Combining linear and non-linear structural information can overcome the drawback of homologous information and enhance the robustness of the obtained structural information in image super-resolution. Taking into account the local minimum of gradient descent, a cosine annealing mechanism is used to optimize the training procedure by performing warm restarts and adjusting the learning rate. Experimental results illustrate that the proposed CSRNet is competitive with state-of-the-art methods in image super-resolution.

An effective method for improving the object detection performance is to decrease the number of false positive (NFP) detection boxes and increase the number of true positive (NTP) detection boxes. In terms of the region-based object detection framework, an appropriate sample weighting strategy can help effectively achieve this goal without causing any inference efficiency loss. However, designing a suitable weighting method is not easy, and a reasonable guiding metric and comprehensive analysis are needed. This article directly sets the NFP and NTP as the evaluation metrics and examines how some preliminary weighting methods affect these two metrics. Based on the results of our analysis, we carefully design a simple yet effective sample weighting method, referred to as the interval normalization weighting strategy (INWS). Unlike some previous works, which only view sample losses as the weighting factor (e.g., focal losses), the INWS applies both the foreground score and the intersection over union (IoU) as the weighting factors. The INWS consists of two components: the IoU interval score normalization strategy (IISNS) for negative samples and the score interval IoU normalization strategy (SIINS) for positive samples. The IISNS can effectively decrease the NFP, and the SIINS is beneficial for increasing the NTP, especially under higher IoU thresholds. Furthermore, the INWS is convenient for application to most of the existing region-based object detection models. The experimental results on the mainstream benchmarks demonstrate that our INWS can achieve consistent improvements on various baselines.

Dynamic functional brain networks (DFBNs) are powerful tools in neuroscience research. Recent studies reveal that DFBNs contain heterogeneous neural nodes with more extensive connections and more drastic temporal changes, which play pivotal roles in coordinating the reorganization of the brain. Moreover, the spatio-temporal patterns of these nodes are modulated by the brain's historical states. However, existing methods not only ignore the spatio-temporal heterogeneity of neural nodes, but also fail to effectively encode the temporal propagation mechanism of heterogeneous activities. These limitations hinder the deep exploration of spatio-temporal relationships within DFBNs, preventing the capture of abnormal neural heterogeneity caused by brain diseases. To address these challenges, this paper propose a neuro-heterogeneity guided temporal graph learning strategy (NeuroH-TGL). Specifically, we first develop a spatio-temporal pattern decoupling module to disentangle DFBNs into topological consistency networks and temporal trend networks that align with the brain's operational mechanisms. Then, we introduce a heterogeneity mining module to identify pivotal heterogeneity nodes that drive brain reorganization from the two decoupled networks. Finally, we design temporal propagation graph convolution to simulate the influence of the historical states of heterogeneity nodes on the current topology, thereby flexibly extracting heterogeneous spatio-temporal information from the brain. Experiments show that our method surpasses several state-of-the-art methods, and can identify abnormal heterogeneous nodes caused by brain diseases.

Dynamic functional brain network (DFBN) can flexibly describe the time-varying topological connectivity patterns of the brain, and show great potential in brain disease diagnosis. However, most of the existing DFBN analysis methods focus on capturing the dynamic interaction at the brain region level, ignoring the spatio-temporal topological evolution across time windows. Moreover, they are difficult to suppress interfering connections in DFBNs, which leads to a diminished capacity for discerning the intrinsic structures that are intimately linked to brain disorders. To address these issues, we propose a topological evolution graph learning model to capture disease-related spatio-temporal topological features in DFBNs. Specifically, we first take the hubness of adjacent DFBN as the source domain and the target domain in turn, and then use Wasserstein distance (WD) and Gromov-Wasserstein distance (GWD) to capture the brain's evolution law at the node and edge levels, respectively. Furthermore, we introduce the principle of relevant information to guide the topology evolution graph to learn the structures that are most relevant to brain diseases yet least redundant information between adjacent DFBNs. On this basis, we develop a high-order spatio-temporal model with multi-hop graph convolution to collaboratively extract long-range spatial and temporal dependencies from the topological evolution graph. Extensive experiments show that the proposed method outperforms the current state-of-the-art methods, and can effectively reveal the information evolution mechanism between brain regions across windows.

Convolutional neural networks (CNNs) can automatically learn data patterns to express face images for facial expression recognition (FER). However, they may ignore effect of facial segmentation of FER. In this paper, we propose a perception CNN for FER as well as PCNN. Firstly, PCNN can use five parallel networks to simultaneously learn local facial features based on eyes, cheeks and mouth to realize the sensitive capture of the subtle changes in FER. Secondly, we utilize a multi-domain interaction mechanism to register and fuse between local sense organ features and global facial structural features to better express face images for FER. Finally, we design a two-phase loss function to restrict accuracy of obtained sense information and reconstructed face images to guarantee performance of obtained PCNN in FER. Experimental results show that our PCNN achieves superior results on several lab and real-world FER benchmarks: CK+, JAFFE, FER2013, FERPlus, RAF-DB and Occlusion and Pose Variant Dataset. Its code is available at https://github.com/hellloxiaotian/PCNN.

The dynamic functional brain network (DFBN) inherently captures topological changes in brain connectivity pattern during activity, attracting increasing attention for detecting brain disorders. However, most current DFBN analysis methods rely on data-driven modeling and ignore crucial prior knowledge of brain structure and function, resulting in weak interpretability of models. Furthermore, effectively extracting dynamic topological features from DFBN is still a challenging issue, due to its intricate spatio-temporal features coupling. In this paper, we propose an interpretable spatio-temporal tensor graph convolutional network for DFBN analysis. Firstly, by incorporating functional and structural priors into the construction of DBFN, we develop a hierarchical DBFN representation with brain region clustering that effectively captures the spatio-temporal topology among subnetworks. Secondly, we design a tensor graph convolutional network with both intra-graph propagation and inter-graph propagation to simultaneously extract the spatio-temporal features from the hierarchical DFBN. Additionally, we derive a functional subnetwork constraint to enhance the consistency within subnetworks and the differences between subnetworks, which guides the learned features to better reflect the topology prior of the brain network. Finally, self-attention is employed to fuse the learned dynamic topological features of different subnetworks for classification. Experimental results on epilepsy, ADNI and ABIDE datasets demonstrate that our method achieves competitive diagnostic performance and offers network-level interpretability for brain disease diagnosis.

Popular convolutional neural networks mainly use paired images in a supervised way for image watermark removal. However, watermarked images do not have reference images in the real world, which results in poor robustness of image watermark removal techniques. In this paper, we propose a self-supervised convolutional neural network (CNN) in image watermark removal (SWCNN). SWCNN uses a self-supervised way to construct reference watermarked images rather than given paired training samples, according to watermark distribution. A heterogeneous U-Net architecture is used to extract more complementary structural information via simple components for image watermark removal. Taking into account texture information, a mixed loss is exploited to improve visual effects of image watermark removal. Besides, a watermark dataset is conducted. Experimental results show that the proposed SWCNN is superior to popular CNNs in image watermark removal.

Popular methods usually use a degradation model in a supervised way to learn a watermark removal model. However, it is true that reference images are difficult to obtain in the real world, as well as collected images by cameras suffer from noise. To overcome these drawbacks, we propose a perceptive self-supervised learning network for noisy image watermark removal (PSLNet) in this paper. PSLNet depends on a parallel network to remove noise and watermarks. The upper network uses task decomposition ideas to remove noise and watermarks in sequence. The lower network utilizes the degradation model idea to simultaneously remove noise and watermarks. Specifically, mentioned paired watermark images are obtained in a self-supervised way, and paired noisy images (i.e., noisy and reference images) are obtained in a supervised way. To enhance the clarity of obtained images, interacting two sub-networks and fusing obtained clean images are used to improve the effects of image watermark removal in terms of structural information and pixel enhancement. Taking into texture information account, a mixed loss uses obtained images and features to achieve a robust model of noisy image watermark removal. Comprehensive experiments show that our proposed method is very effective in comparison with popular convolutional neural networks (CNNs) for noisy image watermark removal. Codes can be obtained at https://github.com/hellloxiaotian/PSLNet.

Current methods for salient object detection in optical remote sensing images (RSI-SOD) adhere strictly to the conventional supervised train-test paradigm, where models remain fixed after training and are directly applied to test samples. However, this paradigm faces significant challenges in adapting to test-time images due to the inherent variability in remote sensing scenes. Salient objects exhibit considerable differences in size, type, and topology across RSIs, complicating accurate localization in unseen test images. Moreover, the acquisition of RSIs is highly susceptible to atmospheric conditions, often leading to degraded image quality and a notable domain shift between training and testing phases. In this work, we explore test-time model adaptation for RSI-SOD and introduce a novel multitask collaboration approach to tackle these challenges. Our approach integrates a self-supervised auxiliary task, specifically image reconstruction, with the primary supervised task of saliency prediction to achieve collaborative learning. This is accomplished through an architecture that comprises a shared feature encoder and two distinct task-specific decoders. Most importantly, the self-supervised image reconstruction task optimizes model parameters using unlabeled test-time images, allowing adaptation to test distributions and enabling flexibly scene-dependent representation learning. In addition, we design a cross-task modulation module (CMM) positioned between the task-specific decoders, which fully exploits intertask correlations to enhance the adjustment of saliency representations. Extensive experimental evaluations confirm the superiority of our method across three widely used RSI-SOD benchmarks and validate the robustness of our proposed test-time adaptation strategy against diverse types of RSI corruptions.

Current state-of-the-art methods for salient object detection in optical remote sensing images (RSI-SOD) primarily relies on individual image context to detect salient objects. However, the potential of cross-image contexts remains largely unexplored in existing works, which can provide valuable auxiliary and complementary information for discriminating object representations in RSIs. In this paper, we investigate the utilization of cross-image contextual information for RSI-SOD. We propose a novel memory-based context propagation network (MCP-Net) to harness dataset-level contextual information. MCP-Net incorporates a cross-image dual memory module (CDM) to store dataset-level information and utilize it to generate contextual information for the current image. CDM effectively captures intra-scene variations by leveraging both foreground and background memory banks, resulting in improved object representations. Additionally, we enhance the representations by leveraging scale-aware context information within individual images. To preserve RSI details before memory modules, we introduce a shared attention-guided fusion module (SAF) to align the adjacent network-level features. Extensive evaluation results demonstrate the superior performance of our proposed method compared to state-of-the-art methods on three public benchmarks. These results affirm that the inclusion of cross-image contexts can significantly benefit salient object detection in remote sensing images.

Convolutional neural networks (CNNs) have obtained remarkable performance via deep architectures. However, these CNNs often achieve poor robustness for image super-resolution (SR) under complex scenes. In this article, we present a heterogeneous group SR CNN (HGSRCNN) via leveraging structure information of different types to obtain a high-quality image. Specifically, each heterogeneous group block (HGB) of HGSRCNN uses a heterogeneous architecture containing a symmetric group convolutional block and a complementary convolutional block in a parallel way to enhance the internal and external relations of different channels for facilitating richer low-frequency structure information of different types. To prevent the appearance of obtained redundant features, a refinement block (RB) with signal enhancements in a serial way is designed to filter useless information. To prevent the loss of original information, a multilevel enhancement mechanism guides a CNN to achieve a symmetric architecture for promoting expressive ability of HGSRCNN. Besides, a parallel upsampling mechanism is developed to train a blind SR model. Extensive experiments illustrate that the proposed HGSRCNN has obtained excellent SR performance in terms of both quantitative and qualitative analysis. Codes can be accessed at https://github.com/hellloxiaotian/HGSRCNN.

Deep networks can usually depend on extracting more structural information to improve denoising results. However, they may ignore correlation between pixels from an image to pursue better-denoising performance. Window Transformer can use long- and short-distance modeling to interact pixels to address mentioned problem. To make a tradeoff between distance modeling and denoising time, we propose a heterogeneous window Transformer (HWformer) for image denoising. HWformer first designs heterogeneous global windows to capture global context information for improving denoising effects. To build a bridge between long and short-distance modeling, global windows are horizontally and vertically shifted to facilitate diversified information without increasing denoising time. To prevent the information loss phenomenon of independent patches, sparse idea is guided a feed-forward network to extract local information of neighboring patches. The proposed HWformer only takes 30% of popular restoration Transformer in terms of denoising time. Its codes can be obtained athttps://github.com/hellloxiaotian/HWformer.

Although remarkable progress has been made for salient object detection (SOD) in optical remote sensing images (RSIs), the static network design paradigm adopted by existing methods would limit their adaptability to large variations in remote sensing scenes as well as object appearances. In contrast, we explore this research issue from the perspective of generating dynamic network filters in which the parameters are conditioned on specific scene- and location-level contexts. In this paper, we propose a Progressive Context-aware Dynamic Network (PCD-Net) for SOD in RSIs, which adaptively captures context information and adjusts its filtering parameters for saliency detection. PCD-Net adopts an encoder-decoder architecture in which encoded feature representations are progressively decoded by a newly proposed dynamic module, namely Pyramid Scene- and Location-sensitive Dynamic filtering module (PSLD), to generate saliency representations. Furthermore, to transfer effective features from the encoder to the decoder, we construct a Dynamic Transfer Attention (DTA) module to control the interference between the encoder and the decoder in a more flexible way. Extensive evaluations on two commonly-used benchmarks demonstrate the superiority of the proposed method against the existing state-of-the-art methods.

Quantitative trait locus (QTL) analyses of multiomic molecular traits, such as gene transcription (eQTL), DNA methylation (mQTL) and histone modification (haQTL), have been widely used to infer the functional effects of genome variants. However, the QTL discovery is largely restricted by the limited study sample size, which demands higher threshold of minor allele frequency and then causes heavy missing molecular trait-variant associations. This happens prominently in single-cell level molecular QTL studies because of sample availability and cost. It is urgent to propose a method to solve this problem in order to enhance discoveries of current molecular QTL studies with small sample size. In this study, we presented an efficient computational framework called xQTLImp to impute missing molecular QTL associations. In the local-region imputation, xQTLImp uses multivariate Gaussian model to impute the missing associations by leveraging known association statistics of variants and the linkage disequilibrium (LD) around. In the genome-wide imputation, novel procedures are implemented to improve efficiency, including dynamically constructing a reused LD buffer, adopting multiple heuristic strategies and parallel computing. Experiments on various multiomic bulk and single-cell sequencing-based QTL datasets have demonstrated high imputation accuracy and novel QTL discovery ability of xQTLImp. Finally, a C++ software package is freely available at https://github.com/stormlovetao/QTLIMP.

In the originally published version of this manuscript, there was an error in the Funding section; ‘National Natural Science Foundation of China (6210071334, 62072376)’ has now been corrected to ‘National Natural Science Foundation of China (62102319, 62072376)’.

Heterogeneous palmprint recognition has attracted considerable research attention in recent years because it has the potential to greatly improve the recognition performance for personal authentication. In this article, we propose a simultaneous heterogeneous palmprint feature learning and encoding method for heterogeneous palmprint recognition. Unlike existing hand-crafted palmprint descriptors that usually extract features from raw pixels and require strong prior knowledge to design them, the proposed method automatically learns the discriminant binary codes from the informative direction convolution difference vectors of palmprint images. Differing from most heterogeneous palmprint descriptors that individually extract palmprint features from each modality, our method jointly learns the discriminant features from heterogeneous palmprint images so that the specific discriminant properties of different modalities can be better exploited. Furthermore, we present a general heterogeneous palmprint discriminative feature learning model to make the proposed method suitable for multiple heterogeneous palmprint recognition. Experimental results on the widely used PolyU multispectral palmprint database clearly demonstrate the effectiveness of the proposed method.

Deep convolutional neural networks (CNNs) have been widely applied for low-level vision over the past five years. According to the nature of different applications, designing appropriate CNN architectures is developed. However, customized architectures gather different features via treating all pixel points as equal to improve the performance of given application, which ignores the effects of local power pixel points and results in low training efficiency. In this article, we propose an asymmetric CNN (ACNet) comprising an asymmetric block (AB), a memory enhancement block (MEB), and a high-frequency feature enhancement block (HFFEB) for image superresolution (SR). The AB utilizes one-dimensional (1-D) asymmetric convolutions to intensify the square convolution kernels in horizontal and vertical directions for promoting the influences of local salient features for single image SR (SISR). The MEB fuses all hierarchical low-frequency features from AB via a residual learning technique to resolve the long-term dependency problem and transforms obtained low-frequency features into high-frequency features. The HFFEB exploits low- and high-frequency features to obtain more robust SR features and address the excessive feature enhancement problem. Additionally, it also takes charge of reconstructing a high-resolution image. Extensive experiments show that our ACNet can effectively address SISR, blind SISR, and blind SISR of blind noise problems. The code of the ACNet is shown athttps://github.com/hellloxiaotian/ACNet.

Cross-modal hashing has received increasing research attentions due to its less storage and efficient retrieval. However, most existing cross-modal hashing methods focus only on exploring multi-modal information, while underestimate the significance of local and Euclidean structure information on the hashing learning procedure. In this paper, we propose a supervised discrete-based cross-modal hashing method, named Scalable Discriminative Discrete Hashing (SDDH), for cross-modal retrieval, where 1) the discrete hash codes are directly obtained by multi-modal features and semantic labels so that the quantization errors are dramatically reduced, and 2) the discrete hash codes simultaneously preserve the heterogeneous similarity and manifold information in the original space by employing matrix factoring with orthogonal and balanced constraints. Moreover, an efficient optimization is introduced to tackle the discrete solution, which makes the SDDH scalable to large-scale cross-modal retrieval. Empirical results on three widely-used benchmark databases clearly demonstrate the effectiveness and efficiency of the proposed method in comparison with state-of-the-arts.

Deep convolutional neural networks (CNNs) have been popularly adopted in image super-resolution (SR). However, deep CNNs for SR often suffer from the instability of training, resulting in poor image SR performance. Gathering complementary contextual information can effectively overcome the problem. Along this line, we propose a coarse-to-fine SR CNN (CFSRCNN) to recover a high-resolution (HR) image from its low-resolution version. The proposed CFSRCNN consists of a stack of feature extraction blocks (FEBs), an enhancement block (EB), a construction block (CB) and, a feature refinement block (FRB) to learn a robust SR model. Specifically, the stack of FEBs learns the long- and short-path features, and then fuses the learned features by expending the effect of the shallower layers to the deeper layers to improve the representing power of learned features. A compression unit is then used in each FEB to distill important information of features so as to reduce the number of parameters. Subsequently, the EB utilizes residual learning to integrate the extracted features to prevent from losing edge information due to repeated distillation operations. After that, the CB applies the global and local LR features to obtain coarse features, followed by the FRB to refine the features to reconstruct a high-resolution image. Extensive experiments demonstrate the high efficiency and good performance of our CFSRCNN model on benchmark datasets compared with state-of-the-art SR models. The code of CFSRCNN is accessible onhttps://github.com/hellloxiaotian/CFSRCNN.

3D palmprint-based biometric recognition has drawn growing research attention due to its several merits over 2D counterpart such as robust structural measurement of a palm surface and high anti-counterfeiting capability. However, most existing 3D palmprint descriptors are hand-crafted that usually extract stationary features from 3D palmprint images. In this paper, we propose a feature learning method to jointly learn compact curvature feature descriptor for 3D palmprint recognition. We first form multiple curvature data vectors to completely sample the intrinsic curvature information of 3D palmprint images. Then, we jointly learn a feature projection function that project curvature data vectors into binary feature codes, which have the maximum inter-class variances and minimum intra-class distance so that they are discriminative. Moreover, we learn the collaborative binary representation of the multiple curvature feature codes by minimizing the information loss between the final representation and the multiple curvature features, so that the proposed method is more compact in feature representation and efficient in matching. Experimental results on the baseline 3D palmprint database demonstrate the superiority of the proposed method in terms of recognition performance in comparison with state-of-the-art 3D palmprint descriptors.

Direction information has been intensively investigated for Finger-Knuckle-Print (FKP) recognition. However, most existing direction-based KFP recognition methods are handcrafted, which are heuristic and require too much prior knowledge to engineer them. In this paper, we propose a discriminative direction binary feature learning (DDBFL) method for FKP recognition. We first propose a direction convolution difference vector (DCDV) to better describe the direction information of FKP images. Then, we learn a feature projection to convert the DCDV into binary codes, which are compact for the intra-class samples and more separable for the inter-class samples. Finally, we concatenate the block-wise histograms of the DDBFL codes to form the final descriptor for FKP recognition. Experimental results on the baseline PolyU FKP database demonstrate the competitive performance of the proposed method.