Functional Magnetic Resonance Image (fMRI) is commonly employed to study human brain activity, since it offers insight into the relationship between functional fluctuations and human behavior. To enhance analysis and comprehension of brain activity, Graph Neural Networks (GNNs) have been widely applied to the analysis of functional connectivities (FC) derived from fMRI data, due to their ability to capture the synergistic interactions among brain regions. However, in the human brain, performing complex tasks typically involves the activation of certain pathways, which could be represented as paths across graphs. As such, conventional GNNs struggle to learn from these pathways due to the long-range dependencies of multiple pathways. To address these challenges, we introduce a novel framework BrainMAP to learn multiple pathways in brain networks. BrainMAP leverages sequential models to identify long-range correlations among sequentialized brain regions and incorporates an aggregation module based on Mixture of Experts (MoE) to learn from multiple pathways. Our comprehensive experiments highlight BrainMAP's superior performance. Furthermore, our framework enables explanatory analyses of crucial brain regions involved in tasks.

Multimodal neuroimaging data modeling has become a widely used approach but confronts considerable challenges due to their heterogeneity, which encompasses variability in data types, scales, and formats across modalities. This variability necessitates the deployment of advanced computational methods to integrate and interpret diverse datasets within a cohesive analytical framework. In our research, we combine functional magnetic resonance imaging (fMRI), diffusion tensor imaging (DTI), and structural MRI (sMRI) for joint analysis. This integration capitalizes on the unique strengths of each modality and their inherent interconnections, aiming for a comprehensive understanding of the brain's connectivity and anatomical characteristics. Utilizing the Glasser atlas for parcellation, we integrate imaging-derived features from multiple modalities-functional connectivity from fMRI, structural connectivity from DTI, and anatomical features from sMRI-within consistent regions. Our approach incorporates a masking strategy to differentially weight neural connections, thereby facilitating an amalgamation of multimodal imaging data. This technique enhances interpretability at the connectivity level, transcending traditional analyses centered on singular regional attributes. The model is applied to the Human Connectome Project's Development study to elucidate the associations between multimodal imaging and cognitive functions throughout youth. The analysis demonstrates improved prediction accuracy and uncovers crucial anatomical features and neural connections, deepening our understanding of brain structure and function. This study not only advances multimodal neuroimaging analytics by offering a novel method for integrative analysis of diverse imaging modalities but also improves the understanding of intricate relationships between brain's structural and functional networks and cognitive development.

Recent advances in imaging genetics make it possible to combine different types of data including medical images like functional magnetic resonance imaging (fMRI) and genetic data like single nucleotide polymorphisms (SNPs) for comprehensive diagnosis of mental disorders. Understanding complex interactions among these heterogeneous data may give rise to a new perspective, while at the same time demand statistical models for their integration. Various graphical models have been proposed for the study of interaction or association networks with continuous, binary, and count data as well as the mixture of them. However, limited efforts have been made for the multinomial case, for instance, SNP data. Our goal is therefore to fill the void by developing a graphical model for the integration of fMRI image and SNP data, which can provide deeper understanding of the unknown neurogenetic mechanism. In this article, we propose a latent Gaussian copula model for mixed data containing multinomial components. We assume that the discrete variable is obtained by discretizing a latent (unobserved) continuous variable and then create a semi-rank based estimator of the graph structure. The simulation results demonstrate that the proposed latent correlation has more steady and accurate performance than several existing methods in detecting graph structure. When applying to a real schizophrenia data consisting of SNP array and fMRI image collected by the Mind Clinical Imaging Consortium (MCIC), the proposed method reveals a set of distinct SNP-brain associations, which are verified to be biologically significant. The proposed model is statistically promising in handling mixed types of data including multinomial components, which can find widespread applications. To promote reproducible research, the R code is available at https://github.com/Aiying0512/LGCM.

The combination of multimodal imaging and genomics provides a more comprehensive way for the study of mental illnesses and brain functions. Deep network-based data fusion models have been developed to capture their complex associations, resulting in improved diagnosis of diseases. However, deep learning models are often difficult to interpret, bringing about challenges for uncovering biological mechanisms using these models. In this work, we develop an interpretable multimodal fusion model to perform automated diagnosis and result interpretation simultaneously. We name it Grad-CAM guided convolutional collaborative learning (gCAM-CCL), which is achieved by combining intermediate feature maps with gradient-based weights. The gCAM-CCL model can generate interpretable activation maps to quantify pixel-level contributions of the input features. Moreover, the estimated activation maps are class-specific, which can therefore facilitate the identification of biomarkers underlying different groups. We validate the gCAM-CCL model on a brain imaging-genetic study, and demonstrate its applications to both the classification of cognitive function groups and the discovery of underlying biological mechanisms. Specifically, our analysis results suggest that during task-fMRI scans, several object recognition related regions of interests (ROIs) are activated followed by several downstream encoding ROIs. In addition, the high cognitive group may have stronger neurotransmission signaling while the low cognitive group may have problems in brain/neuron development due to genetic variations.

Adolescence is a transitional period between the childhood and adulthood with physical changes, as well as increasing emotional development. Studies have shown that the emotional sensitivity is related to a second period of rapid brain growth. However, there is little focus on the trend of brain development during this period. In this paper, we aim to track functional brain connectivity development from late childhood to young adulthood. Mathematically, this problem can be modeled via the estimation of multiple Gaussian graphical models (GGMs). However, most existing methods either require the graph sequence to be fairly long or are only applicable to small graphs. In this paper, we adapted a Bayesian approach incorporating joint estimation of multiple GGMs to overcome the short sequence difficulty, which is also computationally efficient. The data used are the functional magnetic resonance imaging (fMRI) images obtained from the publicly available Philadelphia Neurodevelopmental Cohort (PNC). They include 855 individuals aged 8-22 years who were divided into five different adolescent stages. We summarized the networks with global measurements and applied a hypothesis test across age groups to detect the developmental patterns. Three patterns were detected and defined as consistent development, late puberty, and temporal change. We also discovered several anatomical areas, such as the middle frontal gyrus, putamen gyrus, right lingual gyrus, and right cerebellum crus 2 that are highly involved in the brain functional development. The functional networks, including the salience, subcortical, and auditory networks are significantly developing during the adolescent period.

Estimating dynamic functional network connectivity (dFNC) of the brain from functional magnetic resonance imaging (fMRI) data can reveal both spatial and temporal organization and can be applied to track the developmental trajectory of brain maturity as well as to study mental illness. Resting state fMRI (rs-fMRI) is regarded as a promising task since it reflects the spontaneous brain activity without an external stimulus. The sliding window method has been successfully used to extract dFNC but typically assumes a fixed window size. The hidden Markov model (HMM) based method is an alternative approach for estimating time-varying connectivity. In this paper, we propose a sparse HMM based on Gaussian HMM and Gaussian graphical model (GGM). In this model, the time-varying neural processes are represented as discrete brain states which are described with functional connectivity networks. By enforcing the sparsity on the precision matrix, we can get interpretable connectivity between different functional regions. The optimization of our model can be realized with the expectation maximization (EM) and graphical least absolute shrinkage and selection operator (glasso) algorithms. The proposed model is validated on both simulated blood oxygenation-level dependent (BOLD) time series and rs-fMRI data. Results indicate that the proposed model can capture both stationary and abrupt brain activity fluctuations. We also compare dFNC patterns between children and young adults from the Philadelphia Neurodevelopmental Cohort (PNC) study. Both spatial and temporal behavior of the dFNC are analyzed and compared. The results provide insight into the developmental trajectory across childhood and motivate further research on brain connectivity.

Schizophrenia (SZ) is a chronic and severe mental disorder that affects how a person thinks, feels, and behaves. It has been proposed that this disorder is related to disrupted brain connectivity, which has been verified by many studies. With the development of functional magnetic resonance imaging (fMRI), further exploration of brain connectivity was made possible. Region-based networks are commonly used for mapping brain connectivity. However, they fail to illustrate the connectivity within regions of interest (ROIs) and lose precise location information. Voxel-based networks provide higher precision, but are difficult to construct and interpret due to the high dimensionality of the data. In this paper, we adopt a novel high-dimensional Gaussian graphical model - ψ-learning method, which can help ease computational burden and provide more accurate inference for the underlying networks. This method has been proven to be an equivalent measure of the partial correlation coefficient and, thus, is flexible for network comparison through statistical tests. The fMRI data we used were collected by the mind clinical imaging consortium using an auditory task in which there are 92 SZ patients and 116 healthy controls. We compared the networks at three different scales by using global measurements, community structure, and edge-wise comparisons within the networks. Our results reveal, at the highest voxel resolution, sets of distinct aberrant patterns for the SZ patients, and more precise local structures are provided within ROIs for further investigation.

In addition to the strong correlation in the internal pixels of the image, the pixels in the two phases of image have a certain correlation, that is, temporal context information. Conditional Random Field (CRF) models not only model spatial context information, but also fuse temporal context information. A spatio-temporal model based on temporal context information and spatial context information is proposed to improve the classification accuracy of remote sensing images. Use of two-phase data, the temporal context information between two phases and the spatial context information between pixels within a single temporal phase are considered, and High-order CRF model the spatio-temporal context information. Make full use of spatio-temporal context information of two-phase image to improve the classification accuracy. The experimental results show that the proposed method has better classification accuracy than the classification accuracy without temporal context information.

This paper proposes a new method to fusion pixel-based classifier and object-based classifier to land cover classification. We choose the Boosting classifier as pixel-based classifier and choose the SVM classifier as object-based classifier. At first, one scene image is classified using Boosting classifier to acquire the labels of each pixel point in the image. Secondly, the same scene image is segmented, and then we cast a vote to each segmentation block, and select the label of the highest votes as the label of the segmentation block. Thirdly, the results of vote and the classification results of SVM classifier are fusion. By we apply the method to Landsat TM, ZiYuan3 and IKONOS images for land cover classification, compare the results of new approach with the results of only using the Boosting algorithms and only using the SVM algorithms. Experimental results show that the significant improvement in classification accuracy.

Object-based classification approach offers higher accuracy and smoother classification map over pixel-based classification, especially on high spatial resolution imagery. However, a potential limitation in using object-based classification is the possible negative impact of under-segmentation. Under-segmentation error cannot be adjusted in the unit of object, and can affect the potential accuracy of classification On the contrary, pixel-based contextual classification utilizes spatial information to reduce salt-and-pepper noise, and does not deal with segmentation issues. Though, a challenge of pixel-based contextual classification is to define appropriate contextual neighbors. In this paper, we propose a new approach, which can avoid the disadvantages of these two kinds of methods, combine the advantage of object-based methods and pixel-based methods, and make a tradeoff between these two methods. We validate the combined approach using two sets of airborne different spatial resolution imagery: TM data and Ziyuan3 image. The results indicate that the proposed method generated highest classification accuracy and best visual effect for the classification map. This study demonstrated the potential of the sub-block approach.