The use of imaging and genetic data for biomarker detection and disease diagnosis can deepen the understanding of disease pathogenesis and assist in clinical diagnosis. However, current methods face two major challenges: 1) the significant heterogeneity between multimodal data hampers modality fusion and 2) effectively exploring consistency and variability information from similar diseases for enhancing model performance is difficult. In this paper, we propose a novel unified framework, termed dual adaptive disentangled representation learning (DADRL), to simultaneously achieve disease-shared and disease-specific biomarker detection as well as disease diagnosis. Our DADRL comprises three components: 1) a biology information constraints-based modality fusion strategy is applied to adaptively explore inter- and intra-modal correlations, thereby effectively fusing multimodal data; 2) a unified framework that integrates modality fusion and disease diagnosis is proposed to mine disease-related information for simultaneously accomplishing disease-related biomarker detection and disease diagnosis; and 3) disentangled representation learning and several adaptive metric constraints are incorporated into the unified framework to adaptively separate disease-specific information from disease-shared feature representations for effectively identifying disease-shared and disease-specific biomarkers, thereby deepening the understanding of disease pathogenesis. Extensive experiments on multiple real datasets and simulated data demonstrate that our method significantly improves performance of biomarker detection and disease diagnosis.

Remote sensing image-text retrieval (RSITR) is a cross-modal task that integrates visual and textual information, attracting significant attention in remote sensing research. Remote sensing images typically contain complex scenes with abundant details, presenting significant challenges for accurate semantic alignment between images and texts. Despite advances in the field, achieving precise alignment in such intricate contexts remains a major hurdle. To address this challenge, this article introduces a novel context-aware local-global semantic alignment (CLGSA) method. The proposed method consists of two key modules: the local key feature alignment (LKFA) module and the cross-sample global semantic alignment (CGSA) module. The LKFA module incorporates a local image masking and reconstruction task to improve the alignment between image and text features. Specifically, this module masks certain regions of the image and uses text context information to guide the reconstruction of the masked areas, enhancing the alignment of local semantics and ensuring more accurate retrieval of region-specific content. The CGSA module employs a hard sample triplet loss to improve global semantic consistency. By prioritizing difficult samples during training, this module refines feature space distributions, helping the model better capture global semantics across the entire image-text pair. A series of extensive experiments demonstrates the effectiveness of the proposed method. The method achieves an mR score of 32.07% on the RSICD dataset and 46.63% on the RSITMD dataset, outperforming baseline methods and confirming the robustness and accuracy of the approach.

The remote sensing image–text retrieval (RSITR) aims to establish semantic alignment between images and texts to enable accurate cross-modal retrieval. Existing methods usually extract features from images and texts independently, aligning them in a shared embedding space to achieve cross-modal retrieval. However, these methods often assume complete alignment between image and text pairs, overlooking the inherent disparities between the rich visual details in remote sensing (RS) images and the abstract nature of textual descriptions. These disparities result in image–text pairs only sharing partial semantic correlations, rather than one-to-one complete alignment. Such incomplete alignment adversely affects model training and retrieval accuracy. To address this problem, a relevance-guided adaptive learning (RGAL) method is proposed, which quantifies and leverages the relevance of image–text pairs to refine the training process while enhancing retrieval performance. First, the proposed method introduces an image–text relevance measurement mechanism that integrates global and local feature distances to accurately evaluate the degree of semantic relevance between images and texts. Second, a relevance-based sample division (RBSD) strategy is proposed, utilizing a Gaussian mixture model to dynamically redivide samples into positive and negative pairs according to the measured image–text relevance. This strategy refines the training dataset, reduces noise, and enhances the effectiveness of model learning. Finally, a relevance-weighted triplet loss (RWTL) is designed to adaptively adjust the contribution of sample pairs to the loss function based on their relevance, further optimizing model training and enhancing retrieval accuracy. Experimental results on multiple RSITR datasets demonstrate that the proposed method significantly improves retrieval accuracy and performance.

Visible-infrared person re-identification (VI-ReID) task aims to retrieve persons from different spectrum cameras (i.e., visible and infrared images). The biggest challenge of VI-ReID is the huge cross-modal discrepancy caused by different imaging mechanisms. Many VI-ReID methods have been proposed by embedding different modal person images into a shared feature space to narrow the cross-modal discrepancy. However, these methods ignore the purification of identity features, which results in identity features containing different modal information and failing to align well. In this article, an identity feature disentanglement method is proposed to disentangle the identity features from identity-irrelevant information, such as pose and modality. Specifically, images of different modalities are first processed to extract shared features that reduce the cross-modal discrepancy preliminarily. Then the extracted feature of each image is disentangled into a latent identity variable and an identity-irrelevant variable. In order to enforce the latent identity variable to contain as much identity information as possible and as little identity-irrelevant information, an ID-discriminative loss and an ID-swapping reconstruction process are additionally designed. Extensive quantitative and qualitative experiments on two popular public VI-ReID datasets, RegDB and SYSU-MM01, demonstrate the efficacy and superiority of the proposed method.

Remote sensing scene classification (RSSC) attempts to label an image with a specific scene category. Recently, convolutional neural networks (CNNs) have shown the powerful feature extraction capability to combine local and global features. However, both the local and global features are extracted independently, which ignore the complementary representation. In this letter, a local–global mutual learning (LML) method is proposed to capture both the global and local features. Specifically, local regions are first generated by highlighting the semantic areas in the corresponding original image. Then, a two-branch architecture is used to extract features for the local regions and global image, respectively. Both the classification loss and mutual learning loss are exploited to train the local–global branches simultaneously, which constrain the two branches to promote each other. Experiments on two popular datasets demonstrate the effectiveness of the proposed method.

Change detection (CD) aims to identify the differences between multitemporal images acquired over the same geographical area at different times. With the advantages of requiring no cumbersome labeled change information, unsupervised CD has attracted extensive attention of researchers. Multitemporal images tend to have different resolutions as they are usually captured at different times with different sensor properties. It is difficult to directly obtain one pixelwise change map for two images with different resolutions, so current methods usually resize multitemporal images to a unified size. However, resizing operations change the original information of pixels, which limits the final CD performance. This article aims to detect changes from multitemporal images in the originally different resolutions without resizing operations. To achieve this, a cross-resolution difference learning method is proposed. Specifically, two cross-resolution pixelwise difference maps are generated for the two different resolution images and fused to produce the final change map. First, the two input images are segmented into individual homogeneous regions separately due to different resolutions. Second, each pixelwise difference map is produced according to two measure distances, the mutual information distance and the deep feature distance, between image regions in which the pixel lies. Third, the final binary change map is generated by fusing and binarizing the two cross-resolution difference maps. Extensive experiments on four datasets demonstrate the effectiveness of the proposed method for detecting changes from different resolution images.

Visible-infrared person re-identification (VI-ReID) task aims to retrieve the same person between visible and infrared images. VI-ReID is challenging as the images captured by different spectra present large cross-modality discrepancy. Many methods adopt a two-stream network and design additional constraint conditions to extract shared features for different modalities. However, the interaction between the feature extraction processes of different modalities is rarely considered. In this paper, a partially interactive collaboration method is proposed to exploit the complementary information of different modalities to reduce the modality gap for VI-ReID. Specifically, the proposed method is achieved in a partially interactive-shared architecture: collaborative shallow layers and shared deep layers. The collaborative shallow layers consider the interaction between modality-specific features of different modalities, encouraging the feature extraction processes of different modalities constrain each other to enhance feature representations. The shared deep layers further embed the modality-specific features to a common space to endow them the same identity discriminability. To ensure the interactive collaborative learning implement effectively, the conventional loss and collaborative loss are utilized jointly to train the whole network. Extensive experiments on two publicly available VI-ReID datasets verify the superiority of the proposed PIC method. Specifically, the proposed method achieves a rank-1 accuracy of 83.6% and 57.5% on RegDB and SYSU-MM01 datasets, respectively.

Person re-identification (ReID) task aims to retrieve the same person across multiple spatially disjoint camera views. Due to huge image changes caused by various factors such as posture variation and illumination transformation, images of different persons may share the more similar appearances than images of the same one. Learning discriminative representations to distinguish details of different persons is significant for person ReID. Many existing methods learn discriminative representations resorting to a human body part location branch which requires cumbersome expert human annotations or complex network designs. In this article, a novel bidirectional interaction network is proposed to explore discriminative representations for person ReID without any human body part detection. The proposed method regards multiple convolutional features as responses to various body part properties and exploits the inter-layer interaction to mine discriminative representations for person identities. Firstly, an inter-layer bilinear pooling strategy is proposed to feasibly exploit the pairwise feature relations between two convolution layers. Secondly, to explore interaction of multiple layers, an effective bidirectional integration strategy consisting of two different multi-layer interaction processes is designed to aggregate bilinear pooling interaction of multiple convolution layers. The interaction of multiple layers is implemented in a layer-by-layer nesting policy to ensure the two interaction processes are different and complementary. Extensive experiments validate the superiority of the proposed method on four popular person ReID datasets including Market-1501, DukeMTMC-ReID, CUHK03-NP and MSMT17. Specifically, the proposed method achieves a rank-1 accuracy of 95.1% and 88.2% on Market-1501 and DukeMTMC-ReID, respectively.

Imaging genetics is an effective tool used to detect potential biomarkers of Alzheimer's disease (AD) in imaging and genetic data. Most existing imaging genetics methods analyze the association between brain imaging quantitative traits (QTs) and genetic data [e.g., single nucleotide polymorphism (SNP)] by using a linear model, ignoring correlations between a set of QTs and SNP groups, and disregarding the varied associations between longitudinal imaging QTs and SNPs. To solve these problems, we propose a novel temporal group sparsity regression and additive model (T-GSRAM) to identify associations between longitudinal imaging QTs and SNPs for detection of potential AD biomarkers. We first construct a nonparametric regression model to analyze the nonlinear association between QTs and SNPs, which can accurately model the complex influence of SNPs on QTs. We then use longitudinal QTs to identify the trajectory of imaging genetic patterns over time. Moreover, the SNP information of group and individual levels are incorporated into the proposed method to boost the power of biomarker detection. Finally, we propose an efficient algorithm to solve the whole T-GSRAM model. We evaluated our method using simulation data and real data obtained from AD neuroimaging initiative. Experimental results show that our proposed method outperforms several state-of-the-art methods in terms of the receiver operating characteristic curves and area under the curve. Moreover, the detection of AD-related genes and QTs has been confirmed in previous studies, thereby further verifying the effectiveness of our approach and helping understand the genetic basis over time during disease progression.

This paper studies the task of 3D human pose estimation from a single RGB image, which is challenging without depth information. Recently many deep learning methods are proposed and achieve great improvements due to their strong representation learning. However, most existing methods ignore the relationship between joint features. In this paper, a joint relationship aware neural network is proposed to take both global and local joint relationship into consideration. First, a whole feature block representing all human body joints is extracted by a convolutional neural network. A Dual Attention Module (DAM) is applied on the whole feature block to generate attention weights. By exploiting the attention module, the global relationship between the whole joints is encoded. Second, the weighted whole feature block is divided into some individual joint features. To capture salient joint feature, the individual joint features are refined by individual DAMs. Finally, a joint angle prediction constraint is proposed to consider local joint relationship. Quantitative and qualitative experiments on 3D human pose estimation benchmarks demonstrate the effectiveness of the proposed method.