Multimodal neuroimaging provides complementary structural and functional insights into both human brain organization and disease-related dynamics. Recent studies demonstrate enhanced diagnostic sensitivity for Alzheimer’s disease (AD) through synergistic integration of neuroimaging data (e.g., sMRI, fMRI) with tabular data (e.g., behavioral and cognitive tests). However, the intrinsic heterogeneity across modalities (e.g., 4D spatiotemporal fMRI dynamics vs. 3D anatomical sMRI structure) presents critical challenges for discriminative feature fusion, often leading to information loss or biased fusion. To bridge this gap, we propose M2M-AlignNet: a multimodal co-attention network with latent alignment for early AD diagnosis using sMRI and fMRI. At the core of our approach is a multi-patch-to-multi-patch (M2M) contrastive loss function that quantifies and reduces representational discrepancies via weighted patch correspondence, explicitly aligning fMRI components across brain regions with their sMRI structural substrates without one-to-one constraints. Additionally, we propose a latent-as-query co-attention module to autonomously discover fusion patterns, circumventing modality prioritization biases while minimizing feature redundancy. We conduct extensive experiments to confirm the effectiveness of our method and highlight the correspondence between fMRI and sMRI as AD biomarkers.

Group independent component analysis (ICA) has been extensively used to extract brain functional networks (FNs) and associated neuroimaging measures from multi-subject functional magnetic resonance imaging (fMRI) data. However, the inherent noise in fMRI data can adversely affect the performance of ICA, often leading to noisy FNs and hindering the identification of network-level biomarkers. To address this challenge, we propose a novel method called group information guided smooth independent component analysis (GIG-sICA). Our method effectively generates smoother functional networks with reduced noise and enhanced functional coherence, while preserving intra-subject independence and inter-subject correspondence of FN. Importantly, GIG-sICA is capable of handling different types of noise either separately or in combination. To validate the efficacy of our approach, we conducted comprehensive experiments, comparing GIG-sICA with traditional group ICA methods on both simulated and real fMRI datasets. Experiments on five simulated datasets, generated by adding various types of noise, demonstrate that GIG-sICA produces smoother functional networks with enhanced spatial accuracy. Additionally, experiments on real fMRI data from 137 schizophrenia patients and 144 healthy controls demonstrate that GIG-sICA more effectively captures functionally meaningful brain networks and reveals clearer group differences. Overall, GIG-sICA produces smooth and precise network estimations, supporting the discovery of robust biomarkers at the network level for neuroscience research.

Dynamic functional connectivity (dFC) analysis investigates how the functional interactions between brain regions change over time by identifying recurring connectivity patterns, known as dFC states, and tracking transitions between them. Non-negative matrix factorization (NMF) has been used in dFC analysis because it produces non-negative dFC states and coefficients, interpreting dFC states and their transitions straightforwardly. However, existing NMF-based methods are limited to processing dFC data with exclusively positive values, failing to align with the functional correlations and anti-correlations between brain regions. This paper proposes an orthogonal semi-nonnegative matrix factorization (OSemiNMF) method, extending NMF to directly handle mixed-sign dFC data. Furthermore, an orthogonality constraint on the bases (i.e., dFC states) is incorporated to enhance the uniqueness of dFC states. For 10 simulated datasets with varying properties, our method outperforms comparison methods, supporting its superior ability to capture dFC states and state transitions. Using four resting-state fMRI datasets consisting of 708 healthy controls (HCs) and 537 schizophrenia patients (SZs), our method identifies reproducible dFC states and state transitions across datasets. Further, our findings reveal that SZs spend less time in high-connectivity states compared to HCs. Our study identifies meaningful and reproducible biomarkers of schizophrenia, mainly involving the connectivity associated with the sub-cortical domain. In summary, the OSemiNMF method facilitates the dFC analysis for understanding brain dynamics.

Accurately and preoperatively predicting survival for high-grade gliomas (HGGs) is important for optimizing treatment strategies. Increasing evidence suggests that brain structural and functional connectivity networks derived from advanced magnetic resonance imaging (MRI) are promising predictors for HGG survival. However, advanced MRIs (e.g., diffusion MRI and functional MRI) are generally clinically inaccessible for HGG patients before initiating therapy. To compensate for lack of advanced MRI modalities in brain network studies, in this paper we evaluate the feasibility and performance of predicting HGG survival using exclusively preoperative multi-modal basic structural MRI (sMRI, e.g., T1- and T2-weighted MRI) based brain regional radiomics similarity networks (R2SNs). To this end, we propose a new cooperative multiplex graph neural network (GNN) based multi-modal R2SN integration framework for preoperative HGG survival prediction. First, multi-modal R2SNs are represented by a multiplex network, where each modality-specific R2SN forms one multiplex layer and nodes (i.e., brain regions of interest (ROIs)) are coupled to their replicas across multiplex layers. This facilitates flexible inter-ROI communications both within and between R2SNs. Second, a cooperative GNN is applied to capture intra-modal node feature propagations within each multiplex layer, followed by attention mechanisms used to capture inter-modal node feature interactions across multiplex layers. Finally, a tailored tumor-aware graph pooling is developed to assemble features from the tumor-intersecting ROIs for survival prediction. Extensive experiments on a collected HGG database with three basic sMRI modalities demonstrate the superiority of our method over state-of-the-art baselines in survival stratification. The code is available at https://github.com/ZiLaoTou/TCM-GNN.

In recent years, assisted diagnosis of Alzheimer’s disease (AD) based on brain imaging has become a forefront research area that integrated artificial intelligence. Structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) techniques respectively reveal changes in brain structure and function. Utilizing deep learning technology, common AD features can be learned from a large amount of imaging data to provide technical support for doctors’ diagnostic evaluations. To achieve more precise AD diagnosis, this study proposed an end-to-end network framework. Firstly, the framework divided brain images into 3D brain regions in the form of patches and used an attention CNN for feature extraction. Subsequently, each brain region feature was associated and optimized with a transformer encoder block equipped with multi-head attention mechanism to better represent the brain region interdependencies for AD and normal controls (NC). Finally, through feature fusion under two sub-networks, the method achieved further improvement in diagnostic performance for AD detection. The effectiveness of our method was validated on the ADNI dataset, and patch-based visualization was used to highlight key brain regions of interest in both imaging modalities.

Brain sMRI-based cognitive estimation of dementia states holds significant clinical value, aiding in assessing AD pathological staging and predicting disease progression. Some existing deep learning methods do not fully integrate the characteristics of brain image, which limits the effective representation of brain image and thus weakens its performance in regression and classification tasks. In this work, we propose a neural network that combining with Transformer attention to achieve cognitive estimation. In the proposed network, we design a Transformer block based on the long-range dependency and position encoding capabilities of Transformer to adapt to brain image and effectively combine it in the regression or classification task. We adapt Transformer attention for MRI feature by incorporating 3D positional encoding and 3D convolution. Our method can automatically identify sensitive regions from whole brain sMRI for end-to-end feature learning. Furthermore, it improves attention to AD sensitive biomarkers and improves the performance of regression tasks to jointly predict multiple cognitive scores. Evaluation on ADNI data sets shows promising performance of our method in cognitive scores estimation and AD classification.

MRI(Magnetic Resonance Imaging), as a non-invasive imaging technology, provides rich information at both the structural and functional levels of brain, offering significant support for the screening of Alzheimer's disease (AD). However, due to the large heterogeneity in data format and spatial characteristics between sMRI and fMRI, achieving effective fusion of these two modalities remains a major challenge. To address this issue, we first designed a Transformer attention module incorporating 3D positional encoding to effectively encode 3D sMRI features. Next, we constructed a cascaded transformer module to address the feature encoding of fMRI and the multimodal feature fusion of MRI images from different spatial domains, thereby enhancing the feature representation of both modalities. Additionally, we adopted a multi-layer fused feature integration strategy to enhance the robustness of multimodal features. Visualization analysis is conducted to demonstrate the effectiveness of our method. And our method significantly outperforms single-modality methods using either sMRI or fMRI, exhibiting superior performance in the AD detection task.

Pre-training techniques based on functional connectivity (FC) have demonstrated great potential in brain disease diagnosis. However, previous studies may disrupt the functional information of training data when designing pre-training tasks, and may be limited by biases stemming from single-task learning or insufficient coordination among multiple task components, thereby hindering the acquisition of robust and generalizable feature representations. To address these limitations, we proposed a novel pre-training framework, named Multi-view Contrastive and Masked ROI Learning (CoMa), to learn general representations from healthy datasets through improved learning tasks, with flexible domain-adaptive fine-tuning for downstream tasks. Results showed that the proposed CoMa achieved superior performance across a broad spectrum of diagnostic tasks, significantly outperforming the alternative methods, emphasizing its generalization and effectiveness. Furthermore, the model can further enhance the diagnostic accuracy through task-specific fine-tuning within particular disease domains, indicating its potential for adaptive disease diagnosis. Additionally, we also identified interpretable diagnostic biomarkers for childhood developmental disorders, psychiatric disorders, and neurodegenerative disorders. Overall, the proposed CoMa is instrumental toward the application of fundamental model for disease diagnosis and improves our understanding of underlying mechanisms of common brain disorders.

Positron Emission Tomography (PET) plays a vital role in the diagnosis of Alzheimer's Disease (AD) by revealing the brain's metabolic activity and detecting metabolic abnormalities in the early stages of AD. However, due to significant inter-subject variability in AD imaging presentations, metabolic patterns often differ across individuals, making it challenging for traditional deep learning models to effectively capture such complex and heterogeneous features. Considering that the brain exhibits structural symmetry between hemispheres but often presents pathological asymmetry and regional differences in disease progression, we propose an end-to-end hybrid framework to better model this nonuniform distribution. Our framework integrates 3D patch CNN for local feature encoding and incorporates Transformer Encoders to enhance global semantic representation. Subsequently, a Hemisphere-aware Cross Transformer is employed to fuse inter-hemispheric information, further improving feature discriminability. To enhance robustness under complex feature distributions, we employed a hybrid cross-entropy loss function to optimize the classification task. Experimental results based on the ADNI dataset demonstrate that our method achieves significant improvement in AD diagnosis and the mild cognitive impairment (MCI) conversion task. Importantly, our visualization of key brain regions further confirms the model's attention to ADrelated biomarkers, highlighting its potential to improve early AD diagnosis and clinical application.

Multimodal fusion provides cross-modality information to understand the human brain from different perspectives that may be missed in single modality analysis. Supervised fusion focuses on extracting multimodal patterns related to specific clinical measures by further incorporating a prior interested reference. However, existing supervised fusion methods cannot extract component that have weak correlations with the reference, which may be lost during the optimization process. Here, we propose a reference-guided parallel independent component analysis (RG-PICA) aiming at identifying multimodal covarying features related to interested reference through global optimization. The intra-modality independence, the inter-modality correlation, and the correlation between modalities and the reference are maximized globally. Simulations show that RG-PICA can accurately extract multimodal features correlated with the weak related reference while keeping cross-modality linkage comparing with seven fusion methods. In real data application, RG-PICA reveals co-varying patterns in schizophrenia (SZ) that links with cognition and correlates between modalities. These results demonstrate RG-PICA can jointly optimize for target components that correlate with the reference while keeping cross-modality linkage. This approach can improve the meaningful detection of reliable reference-linked multimodal brain patterns for brain disorders.

Dynamic functional connectivity (DFC) can capture the neural activity changes over time in the brain. Most existing DFC constructions rely on sliding windows, which can be highly impacted by window type and width. In addition, previous methods fail to fully optimize for discriminative spatial-temporal (ST) information and can suffer from inter-site heterogeneity, resulting in suboptimal sensitivity to brain disorders. Here, we propose a novel DFC model by combining ST attention-based bidirectional long short-term memory (BiLSTM) and multi-source domain adaptation (DA) to extract inherent ST information and reduce inter-site heterogeneity. An adaptive similarity sparse representation (SR)-based Kalman filter is proposed to obtain DFC with accurate connectivity strength at each time point. ST attention modules are integrated into BiLSTM to capture discriminative ST features with a maximum mean discrepancy (MMD)-constrained module for multi-source DA. Experimental results show that our method achieves high accuracy (90.67%±2.43%) in discriminating schizophrenia (SZ) from controls, outperforming 7 DA, 6 ST, and 5 DFC models. These results demonstrate the effectiveness of the proposed DFC model, which can be used to investigate multi-site fMRI DFC for the diagnosis of brain disorders.

As a highly interconnected complex network system, the brain exhibits changes in interactions due to common brain disorders. Studying changes in brain network interactions can help us quantitatively analyze functional network patterns and changes in these patterns that are linked to brain disorders. However, relationships between brain regions estimated by most current approaches use a single connectivity that does not fully reflect multiple interactions. Here, we propose a novel functional connectivity (FC) construction method, which can estimate both cooperative and competitive (C-C) relationships between the same regions of interest (ROIs) through improved Ising model. We redefine the Ising dynamic equation to represent pairwise interactions from single to C-C relationships. Results show that the estimated C-C connectivities are normally distributed, with intra-subjects’ (n=970) similarity being consistently and significantly higher than inter-subjects’ similarity across datasets. C-C FCs between occipital, parietal, temporal cortex and the limbic system of schizophrenia (SZ, n=178) are more competitive, while healthy control (HC, n=219) tends to be more cooperative. Group differences in C-C patterns between SZ and HC show significant differences in frontal, parietal and occipital regions. The proposed C-C approach provide new insights into the brain dysfunction in SZ, which can also be applied to investigate other brain disorders.

Tucker decomposition has shown advantages in simultaneously extracting group shared and individual features for studying brain function from multi-subject fMRI data. However, Tucker decomposition of complex-valued fMRI data is challenging, since the data are highly noisy, and imposing sparsity constraints on spatial maps, previously used for denoising magnitude-only fMRI data, may remove signal voxels with low amplitudes. Here we propose a new complex-valued low-rank Tucker decomposition (clrTKD) method to extract principal group and individual components from multi-subject fMRI data, and to denoise spatial components based on the small phase change property at a post-processing stage. We derive the update rules using the alternating direction method of multipliers. Simulated and experimental complex-valued fMRI data are used to evaluate the proposed clrTKD method. Results show that the proposed method can extract more contiguous and vital brain activations such as the anterior cingulate cortex region, compared to Tucker decomposition for magnitude-only fMRI data.

The integration of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provides a powerful approach to explore the brain’s spatial and temporal resolution. This study proposes a multimodal data fusion approach to link simultaneous EEG-fMRI data from multiple subjects using a two-step principal component analysis (PCA) followed by joint independent component analysis (jICA). The proposed method first transforms the two modalities into a unified feature space, preserving the concurrently collected temporal and spatial information, and then applies jICA, followed by back-reconstruction to estimate subject-specific joint source maps and timecourses. The analysis maps group data, enabling efficient group-level inferences on cross-modality covariation. When applied to concurrent resting EEG-fMRI datasets of 55 healthy subjects, we identified a total of 45 resting-state components, capturing prominent multimodal representations of default mode, and secondary somatomotor networks among others. The default mode network was active in both alpha and theta frequency bands, while somatomotor network was active in alpha. We observed that lower frequencies exhibit higher power compared to higher frequencies in their spectra. Further statistical group (old vs young) analysis shows that older adults show stronger functional connectivity in higher cognition, sensorimotor, and triple network regions, which may reflect age-related shifts in sensory integration.

Decoding visual experiences from fMRI offers a powerful avenue to understand human perception and develop advanced brain-computer interfaces. However, current progress often prioritizes maximizing reconstruction fidelity while overlooking interpretability, an essential aspect for deriving neuroscientific insight. To address this gap, we propose MoRE-Brain, a neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction. MoRE-Brain uniquely employs a hierarchical Mixture-of-Experts architecture where distinct experts process fMRI signals from functionally related voxel groups, mimicking specialized brain networks. The experts are first trained to encode fMRI into the frozen CLIP space. A finetuned diffusion model then synthesizes images, guided by expert outputs through a novel dual-stage routing mechanism that dynamically weighs expert contributions across the diffusion process. MoRE-Brain offers three main advancements: First, it introduces a novel Mixture-of-Experts architecture grounded in brain network principles for neuro-decoding. Second, it achieves efficient cross-subject generalization by sharing core expert networks while adapting only subject-specific routers. Third, it provides enhanced mechanistic insight, as the explicit routing reveals precisely how different modeled brain regions shape the semantic and spatial attributes of the reconstructed image. Extensive experiments validate MoRE-Brain’s high reconstruction fidelity, with bottleneck analyses further demonstrating its effective utilization of fMRI signals, distinguishing genuine neural decoding from over-reliance on generative priors. Consequently, MoRE-Brain marks a substantial advance towards more generalizable and interpretable fMRI-based visual decoding.

Multimodal neuroimaging data modeling has become a widely used approach but confronts considerable challenges due to their heterogeneity, which encompasses variability in data types, scales, and formats across modalities. This variability necessitates the deployment of advanced computational methods to integrate and interpret diverse datasets within a cohesive analytical framework. In our research, we combine functional magnetic resonance imaging (fMRI), diffusion tensor imaging (DTI), and structural MRI (sMRI) for joint analysis. This integration capitalizes on the unique strengths of each modality and their inherent interconnections, aiming for a comprehensive understanding of the brain's connectivity and anatomical characteristics. Utilizing the Glasser atlas for parcellation, we integrate imaging-derived features from multiple modalities-functional connectivity from fMRI, structural connectivity from DTI, and anatomical features from sMRI-within consistent regions. Our approach incorporates a masking strategy to differentially weight neural connections, thereby facilitating an amalgamation of multimodal imaging data. This technique enhances interpretability at the connectivity level, transcending traditional analyses centered on singular regional attributes. The model is applied to the Human Connectome Project's Development study to elucidate the associations between multimodal imaging and cognitive functions throughout youth. The analysis demonstrates improved prediction accuracy and uncovers crucial anatomical features and neural connections, deepening our understanding of brain structure and function. This study not only advances multimodal neuroimaging analytics by offering a novel method for integrative analysis of diverse imaging modalities but also improves the understanding of intricate relationships between brain's structural and functional networks and cognitive development.

Resting-state functional magnetic resonance imaging (rsfMRI) is a powerful tool for investigating the relationship between brain function and cognitive processes as it allows for the functional organization of the brain to be captured without relying on a specific task or stimuli. In this paper, we present a novel modeling architecture called BrainRGIN for predicting intelligence (fluid, crystallized and total intelligence) using graph neural networks on rsfMRI derived static functional network connectivity matrices. Extending from the existing graph convolution networks, our approach incorporates a clustering-based embedding and graph isomorphism network in the graph convolutional layer to reflect the nature of the brain sub-network organization and efficient network expression, in combination with TopK pooling and attention-based readout functions. We evaluated our proposed architecture on a large dataset, specifically the Adolescent Brain Cognitive Development Dataset, and demonstrated its effectiveness in predicting individual differences in intelligence. Our model achieved lower mean squared errors and higher correlation scores than existing relevant graph architectures and other traditional machine learning models for all of the intelligence prediction tasks. The middle frontal gyrus exhibited a significant contribution to both fluid and crystallized intelligence, suggesting their pivotal role in these cognitive processes. Total composite scores identified a diverse set of brain regions to be relevant which underscores the complex nature of total intelligence. Our GitHub implementation is publicly available on https://github.com/bishalth01/BrainRGIN/.

Resting-state functional magnetic resonance imaging (rs-fMRI) is a noninvasive technique pivotal for understanding human neural mechanisms of intricate cognitive processes. Most rs-fMRI studies compute a single static functional connectivity matrix across brain regions of interest, or dynamic functional connectivity matrices with a sliding window approach. These approaches are at risk of oversimplifying brain dynamics and lack proper consideration of the goal at hand. While deep learning has gained substantial popularity for modeling complex relational data, its application to uncovering the spatiotemporal dynamics of the brain is still limited. In this study we propose a novel interpretable deep learning framework that learns goal-specific functional connectivity matrix directly from time series and employs a specialized graph neural network for the final classification. Our model, DSAM , leverages temporal causal convolutional networks to capture the temporal dynamics in both low- and high-level feature representations, a temporal attention unit to identify important time points, a self-attention unit to construct the goal-specific connectivity matrix, and a novel variant of graph neural network to capture the spatial dynamics for downstream classification. To validate our approach, we conducted experiments on the Human Connectome Project dataset with 1075 samples to build and interpret the model for the classification of sex group, and the Adolescent Brain Cognitive Development Dataset with 8520 samples for independent testing. Compared our proposed framework with other state-of-art models, results suggested this novel approach goes beyond the assumption of a fixed connectivity matrix, and provides evidence of goal-specific brain connectivity patterns, which opens up potential to gain deeper insights into how the human brain adapts its functional connectivity specific to the task at hand. Our implementation can be found on https://github.com/bishalth01/DSAM . • Multilevel Temporal Feature Extraction: • Utilizes a novel multilevel Temporal Convolutional Network (TCN) adaptation to directly extract temporal features from raw brain activity data. • Shared Temporal Attention for Key Time Points: • Implements shared temporal attention mechanisms to selectively focus on the most informative time points, enhancing model efficiency. • Node-Node Self-Attention for Dynamic Brain Connectivity: • Leverages self-attention mechanisms to dynamically construct a goal-specific brain connectivity matrix, capturing complex inter-node interactions. • ROI-Aware Graph Neural Networks (GNNs): • Introduces ROI-aware GNNs to model spatial brain dynamics, ensuring region-specific contextual learning and improving interpretability.

Multimodal fusion provides multiple benefits over single modality analysis by leveraging both shared and complementary information from different modalities. Notably, supervised fusion enjoys extensive interest for capturing multimodal co-varying patterns associated with clinical measures. A key challenge of brain data analysis is how to handle confounds, which, if unaddressed, can lead to an unrealistic description of the relationship between the brain and clinical measures. Current approaches often rely on linear regression to remove covariate effects prior to fusion, which may lead to information loss, rather than pursue the more global strategy of optimizing both fusion and covariates removal simultaneously. Thus, we propose "CR-mCCAR" to jointly optimize for confounds within a guided fusion model, capturing co-varying multimodal patterns associated with a specific clinical domain while also discounting covariate effects. Simulations show that CR-mCCAR separate the reference and covariate factors accurately. Functional and structural neuroimaging data fusion reveals co-varying patterns in attention deficit/hyperactivity disorder (ADHD, striato-thalamo-cortical and salience areas) and in autism spectrum disorder (ASD, salience and fronto-temporal areas) that link with core symptoms but uncorrelate with age and motion. These results replicate in an independent cohort. Downstream classification accuracy between ADHD/ASD and controls is markedly higher for CR-mCCAR compared to fusion and regression separately. CR-mCCAR can be extended to include multiple targets and multiple covariates. Overall, results demonstrate CR-mCCAR can jointly optimize for target components that correlate with the reference(s) while removing nuisance covariates. This approach can improve the meaningful detection of reliable phenotype-linked multimodal biomarkers for brain disorders.

The subjective nature of diagnosing mental disorders complicates achieving accurate diagnoses. The complex relationship among disorders further exacerbates this issue, particularly in clinical practice where conditions like bipolar disorder (BP) and schizophrenia (SZ) can present similar clinical symptoms and cognitive impairments. To address these challenges, this paper proposes a mutualistic multi-network noisy label learning (MMNNLL) method, which aims to enhance diagnostic accuracy by leveraging neuroimaging data under the presence of potential clinical diagnosis bias or errors. MMNNLL effectively utilizes multiple deep neural networks (DNNs) for learning from data with noisy labels by maximizing the consistency among DNNs in identifying and utilizing samples with clean and noisy labels. Experimental results on public CIFAR-10 and PathMNIST datasets demonstrate the effectiveness of our method in classifying independent test data across various types and levels of label noise. Additionally, our MMNNLL method significantly outperforms state-of-the-art noisy label learning methods. When applied to brain functional connectivity data from BP and SZ patients, our method identifies two biotypes that show more pronounced group differences, and improved classification accuracy compared to the original clinical categories, using both traditional machine learning and advanced deep learning techniques. In summary, our method effectively addresses the possible inaccuracy in nosology of mental disorders and achieves transdiagnostic classification through robust noisy label learning via multi-network collaboration and competition.

Time-series data such as fMRI and MEG carry a wealth of inherent spatio-temporal coupling relationship, and their modeling via deep learning is essential for uncovering biological mechanisms. However, current machine learning models for mining spatio-temporal information usually overlook this intrinsic coupling association, in addition to poor explainability. In this paper, we present an explainable learning framework for spatio-temporal coupling. Specifically, this framework constructs a deep learning network based on spatio-temporal correlation, which can well integrate the time-varying coupled relationships between node representation and inter-node connectivity. Furthermore, it explores spatio-temporal evolution at each time step, providing a better explainability of the analysis results. Finally, we apply the proposed framework to brain dynamic functional connectivity (dFC) analysis. Experimental results demonstrate that it can effectively capture the variations in dFC during brain development and the evolution of spatio-temporal information at the resting state. Two distinct developmental functional connectivity (FC) patterns are identified. Specifically, the connectivity among regions related to emotional regulation decreases, while the connectivity associated with cognitive activities increases. In addition, children and young adults display notable cyclic fluctuations in resting-state brain dFC.

Functional magnetic resonance imaging (fMRI), as a non-invasive method to reveal brain function alterations, frequently yields time series with unequal lengths in real-world scenarios, which may arise from factors such as motion artifacts, participant state, and differing scan protocols. This variability conflicts with the traditional methods relying on isometric inputs, which poses a significant challenge for the downstream applications such as brain age prediction. To address this challenge, we introduced Gaussian Process Regression (GPR) to normalize the length of time series and proposed split-channel residual convolution (SC) and self-attention mechanisms (SA) to perform brain age estimation, called GPR-SCSANet. Results showed that the proposed framework, GPR-SCSANet, is able to fully utilize the inherent information and learn richer feature representations from unequal-length fMRI time courses, which significantly improved the prediction accuracy across 3 brain atlases and 5 prediction models. The results demonstrated the effectiveness and robustness of the proposed GPR-SCSANet, showcasing the potential for broader applications in brain age prediction task.

Federated analysis can help perform large-scale analyses using neuroimaging datasets across various research groups overcoming the limitations of institutional data-sharing policies, privacy or regulatory concerns as it requires no data sharing. In this work, we employ a federated neuromark algorithm to generate independent component analysis (ICA) time courses data from functional magnetic resonance imaging (fMRI) data and feed it to a federated deep neural network (DNN) model to perform classification of schizophrenia patients versus controls.

Federated learning (FL) for distributed data has gained significant attention by enabling model training on local data without transferring it to a central system. While this approach protects sensitive information, risks of data leakage still persist, necessitating the integration of privacy-preserving techniques such as differential privacy. In many FL applications, tasks like exploratory data analysis or tracking and monitoring data that change over time are essential. For these purposes, analysts rely on data visualizations to make decisions or draw conclusions. This vision paper emphasizes the importance of federated privacy-preserving visualization and outlines a general pipeline for its implementation. We discuss the challenges of integrating federated visualizations with differential privacy and demonstrate the feasibility of this approach through examples, such as federated privacy-preserving boxplots, scatterplots, and correlation visualizations in neuroimaging. This highlights the need for further research in this promising field.

The cross-modal synthesis between structural magnetic resonance imaging (sMRI) and functional network connectivity (FNC) is a relatively unexplored area in medical imaging, especially with respect to schizophrenia. This study employs conditional Vision Transformer Generative Adversarial Networks (cViT-GANs) to generate FNC data based on sMRI inputs. After training on a comprehensive dataset that included both individuals with schizophrenia and healthy control subjects, our cViT-GAN model effectively synthesized the FNC matrix for each subject, and then formed a group difference FNC matrix, obtaining a Pearson correlation of 0.73 with the actual FNC matrix. In addition, our FNC visualization results demonstrate significant correlations in particular subcortical brain regions, highlighting the model’s capability of capturing detailed structural-functional associations. This performance distinguishes our model from conditional CNN-based GAN alternatives such as Pix2Pix. Our research is one of the first attempts to link sMRI and FNC synthesis, setting it apart from other cross-modal studies that concentrate on T1- and T2-weighted MR images or the fusion of MRI and CT scans.

To capture different aspects of a complex system, the modeling approach should be able to take these effectively into consideration. Two aspects of the human brain we are quite interested in are its interconnected nature and its dynamism. One modeling approach that can capture these two aspects is based on networks that change with time and go beyond pairwise interactions. Partly because of the size of these temporal high-order networks, analyzing and visualizing them is quite a challenge. In this work, we propose a pipeline based on canonical polyadic (CP) decomposition to analyze high-order networks that are resolved in both time and frequency estimated from resting-state functional magnetic resonance imaging (fMRI) data. We show that we can combine different subjects' information into a common frame of reference for comparison. We also show that different factors provide different patterns that are easy to visualize and interpret. To the best of our knowledge, this is the first work that has proposed a pipeline for analyzing different subjects' brain networks while also incorporating temporal and spectral information about high-order interactions.

With neuroimaging data scientists have gained substantial information of the neuronal underpinning of intelligence. Yet how to integrate multimodal neuronal features effectively in relation to intelligence remains elusive. In this paper, we have developed a reference Canonical Correlation Analysis (RCCA) model that extracts latent, correlated multimodal features while enhancing correlation to a reference of interest. We applied RCCA to gray matter and white matter images from 7874 participants, and compared the derived features with those from Principle Components Analysis (PCA) and sparse CCA (SCCA), in terms of association with intelligence and prediction effectiveness using LASSO regression models. Eight RCCA features explained 10%, 16% and 17% variance of fluid intelligence, crystallized intelligence, and total composite score, respectively, which are similar to the percentage of variance explained by over 100 principle components. SCCA features presented the least variance of intelligence. Our results indicate RCCA model can successfully extract features of interest. The top brain regions that contribute to intelligence include the frontal regions and cingulate gyrus.

Joint blind source separation (JBSS) is a powerful framework for extracting latent sources from multiple datasets while keeping their coherence across multiple linked datasets. Algorithms for JBSS, while offering the capability of improved estimation performance, often incur high computational complexity and hence are not scalable to studies with hundreds or thousands of datasets. In this paper, we propose a simple yet efficient method for source separation that exploits both the correlation among sources within each dataset and across the datasets. The proposed method, named reference-guided component analysis (RGCA), uses source templates as references to (i) guide the separation of sources on each dataset and (ii) establish source dependence and automatically align them across the datasets. In addition, we promote independence among latent sources within each dataset by adding orthogonal constraints on the demixing vectors. The resulting optimization admits an analytic solution that enables extremely fast implementation of RGCA. Our numerical results demonstrate that RGCA obtains competitive performance while having a runtime far superior to other JBSS methods. The proposed method provides a robust and scalable solution to multi-subject functional magnetic resonance imaging (fMRI) studies, enabling joint analysis of thousands of subjects within a few minutes.

Subgroup identification is a fundamental step in precision medicine. Recent research applying data-driven methods such as independent component/vector analysis to multi-subject functional magnetic resonance imaging (fMRI) data has effectively revealed meaningful subgroups. These methods typically focus on single-dimensional information, such as individual functional networks or assuming uniform subgroup structures across networks. Given the complex nature of psychiatric disorders, considering the relationships among subjects across different functional networks can offer valuable insights into diagnostic heterogeneity. We introduce a novel subgroup identification method that leverages multiplex community detection to identify subgroups from multi-subject resting-state fMRI data. The proposed method models subject correlations across functional networks as a multiplex network and identifies common communities across multiple networks and unique communities specific to each functional network. Results from applying the proposed method to 464 psychotic patients show that the identified subgroups exhibit significant group differences on multiple meaningful functional networks as well as the clinical scores, which demonstrate the effectiveness of our method on identifying meaningful subgroups.

Idiopathic generalized epilepsy (IGE) is characterized by cryptogenic etiology and the striatum and cerebellum are recognized as modulators of epileptic network. We collected simultaneous electroencephalogram and functional magnetic resonance imaging data from 145 patients with IGE, 34 of whom recorded interictal epileptic discharges (IEDs) during scanning. In states without IEDs, hierarchical connectivity was performed to search core cortical regions which might be potentially modulated by striatum and cerebellum. Node-node and edge-edge moderation models were constructed to depict direct and indirect moderation effects in states with and without IEDs. Patients showed increased hierarchical connectivity with sensorimotor cortices (SMC) and decreased connectivity with regions in the default mode network (DMN). In the state without IEDs, striatum, cerebellum, and thalamus were linked to weaken the interactions of regions in the salience network (SN) with DMN and SMC. In periods with IEDs, overall increased moderation effects on the interaction between regions in SN and DMN, and between regions in DMN and SMC were observed. The thalamus and striatum were implicated in weakening interactions between regions in SN and SMC. The striatum and cerebellum moderated the cortical interaction among DMN, SN, and SMC in alliance with the thalamus, contributing to the dysfunction in states with and without IEDs in IGE. The current work revealed state-specific modulation effects of striatum and cerebellum on thalamocortical circuits and uncovered the potential core cortical targets which might contribute to develop new clinical neuromodulation techniques.

We introduce an innovative, data-driven topological data analysis (TDA) technique for estimating the state spaces of dynamically changing functional human brain networks at rest. Our method utilizes the Wasserstein distance to measure topological differences, enabling the clustering of brain networks into distinct topological states. This technique outperforms the commonly used k-means clustering in identifying brain network state spaces by effectively incorporating the temporal dynamics of the data without the need for explicit model specification. We further investigate the genetic underpinnings of these topological features using a twin study design, examining the heritability of such state changes. Our findings suggest that the topology of brain networks, particularly in their dynamic state changes, may hold significant hidden genetic information.

Graph convolutional deep learning has emerged as a promising method to explore the functional organization of the human brain in neuroscience research. This paper presents a novel framework that utilizes the gated graph transformer (GGT) model to predict individuals' cognitive ability based on functional connectivity (FC) derived from fMRI. Our framework incorporates prior spatial knowledge and uses a random-walk diffusion strategy that captures the intricate structural and functional relationships between different brain regions. Specifically, our approach employs learnable structural and positional encodings (LSPE) in conjunction with a gating mechanism to efficiently disentangle the learning of positional encoding (PE) and graph embeddings. Additionally, we utilize the attention mechanism to derive multi-view node feature embeddings and dynamically distribute propagation weights between each node and its neighbors, which facilitates the identification of significant biomarkers from functional brain networks and thus enhances the interpretability of the findings. To evaluate our proposed model in cognitive ability prediction, we conduct experiments on two large-scale brain imaging datasets: the Philadelphia Neurodevelopmental Cohort (PNC) and the Human Connectome Project (HCP). The results show that our approach not only outperforms existing methods in prediction accuracy but also provides superior explainability, which can be used to identify important FCs underlying cognitive behaviors.

Schizophrenia (SZ) is a neuropsychiatric disorder that affects millions globally. Current diagnosis of SZ is symptom-based, which poses difficulty due to the variability of symptoms across patients. To this end, many recent studies have developed deep learning methods for automated diagnosis of SZ, especially using raw EEG, which provides high temporal precision. For such methods to be productionized, they must be both explainable and robust. Explainable models are essential to identify biomarkers of SZ, and robust models are critical to learn generalizable patterns, especially amidst changes in the implementation environment. One common example is channel loss during EEG recording, which could be detrimental to classifier performance. In this study, we developed a novel channel dropout (CD) approach to increase the robustness of explainable deep learning models trained on EEG data for SZ diagnosis to channel loss. We developed a baseline convolutional neural network (CNN) architecture and implement our approach as a CD layer added to the baseline (CNN-CD). We then applied two explainability approaches to both models for insight into learned spatial and spectral features and show that the application of CD decreases model sensitivity to channel loss. The CNN and CNN-CD achieved accuracies of 81.9% and 80.9% on testing data, respectively. Furthermore, our models heavily prioritized the parietal electrodes and the a-band, which is supported by existing literature. It is our hope that this study motivates the further development of explainable and robust models and bridges the transition from research to application in a clinical decision support role.

Deep learning methods are increasingly being applied to raw electroencephalography (EEG) data. Relative to traditional machine learning methods, deep learning methods can increase model performance through automated feature extraction. Nevertheless, they are also less explainable. Existing raw EEG explainability methods identify relative feature importance but do not identify how features relate to model predictions. In this study, we combine well-characterized first layer filters with a novel post hoc statistical analysis of the filter activations that linearly relates properties of model activations with predictions. We implement our approach within the context of automated sleep stage classification, finding that the model uncovers positive relationships between waveforms resembling sleep spindles and NREM2, between high frequency filters and Awake samples, and between δ activity and NREM3. Our approach represents a significant step forward for raw EEG explainability and has the potential to provide many insights relating learned features and model predictions in future studies.

As the field of deep learning has grown in recent years, its application to the domain of raw resting-state electroencephalography (EEG) has also increased. Relative to traditional machine learning methods or deep learning methods applied to manually engineered features, there are fewer methods for developing deep learning models on small raw EEG datasets. One potential approach for enhancing deep learning performance, in this case, is the use of transfer learning. While a number of studies have presented transfer learning approaches for manually engineered EEG features, relatively few approaches have been developed for raw resting-state EEG. In this study, we propose a novel EEG transfer learning approach wherein we first train a model on a large publicly available single-channel sleep stage classification dataset. We then use the learned representations to develop a classifier for automated major depressive disorder diagnosis with raw multichannel EEG. Statistical testing reveals that our approach significantly improves the performance of our model (p < 0.05), and we also find that the performance of our approach exceeds that of many previous studies using both engineered features and raw EEG. We further examine how transfer learning affected the representations learned by the model through a pair of explainability analyses, identifying key frequency bands and channels utilized across models. Our proposed approach represents a significant step forward for the domain of raw resting-state EEG classification and has broader implications for use with other electrophysiology and time-series modalities. Importantly, it has the potential to expand the use of deep learning methods across a greater variety of raw EEG datasets and lead to the development of more reliable EEG classifiers.

Discovering components that are shared in multiple datasets, next to dataset-specific features, has great potential for studying the relationships between different subjects or tasks in functional Magnetic Resonance Imaging (fMRI) data. Coupled matrix and tensor factorization approaches have been useful for flexible data fusion, or decomposition to extract features that can be used in multiple ways. However, existing methods do not directly recover shared and dataset-specific components, which requires post-processing steps involving additional hyperparameter selection. In this paper, we propose a tensor-based framework for multi-task fMRI data fusion, using a partially constrained canonical polyadic (CP) decomposition model. Differently from previous approaches, the proposed method directly recovers shared and dataset-specific components, leading to results that are directly interpretable. A strategy to select a highly reproducible solution to the decomposition is also proposed. We evaluate the proposed methodology on real fMRI data of three tasks, and show that the proposed method finds meaningful components that clearly identify group differences between patients with schizophrenia and healthy controls.

Joint blind source separation (JBSS) is a powerful tool for analyzing multiple linked datasets, distinguished by the key ability to exploit cross-dataset dependencies. Despite this ability generally improving overall estimation performance, joint decompositions also incur considerable computational costs, which can lead to intractable problems with hundreds or thousands of datasets. In this paper, we introduce an efficient method for large-scale JBSS by multilinear regression. We consider a model where out of all datasets, only a selected subset are first decomposed to provide regressors that sufficiently estimate sources across all datasets. These regressors define a per-source cost function that naturally extends independent vector analysis (IVA) with a multivariate Gaussian source prior (IVA-G), a powerful formulation for exploiting cross-dataset dependencies. Using simulated and real fMRI data, we demonstrate significant advantages of this method compared with other JBSS methods.

Independent component analysis (ICA) of multi-subject functional magnetic resonance imaging (fMRI) data has proven useful in providing a fully multivariate summary that can be used for multiple purposes. ICA can identify patterns that can discriminate between healthy controls (HC) and patients with various mental disorders such as schizophrenia (Sz). Temporal functional network connectivity (tFNC) obtained from ICA can effectively explain the interactions between brain networks. On the other hand, dictionary learning (DL) enables the discovery of hidden information in data using learnable basis signals through the use of sparsity. In this paper, we present a new method that leverages ICA and DL for the identification of directly interpretable patterns to discriminate between the HC and Sz groups. We use multi-subject resting-state fMRI data from 358 subjects and form subject-specific tFNC feature vectors from ICA results. Then, we learn sparse representations of the tFNCs and introduce a new set of sparse features as well as new interpretable patterns from the learned atoms. Our experimental results show that the new representation not only leads to effective classification between HC and Sz groups using sparse features, but can also identify new interpretable patterns from the learned atoms that can help understand the complexities of mental diseases such as schizophrenia.

Deep learning models can perform as well or better than humans in many tasks, especially vision related. Almost exclusively, these models are used to perform classification or prediction. However, deep learning models are usually of black-box nature, and it is often difficult to interpret the model or the features. The lack of interpretability causes a restrain from applying deep learning to fields such as neuroimaging, where the results must be transparent, and interpretable. Therefore, we present a 'glass-box' deep learning model and apply it to the field of neuroimaging. Our model mixes spatial and temporal dimensions in succession to estimate dynamic connectivity between the brain's intrinsic networks. The interpretable connectivity matrices produced by our model result in beating state-of-the-art models on many tasks using multiple functional MRI datasets. More importantly, our model estimates task-based flexible connectivity matrices, unlike static methods such as Pearson's correlation coefficients.

Identification of subgroups of subjects homogeneous functional networks is a key step for precision medicine. Independent vector analysis (IVA) is shown to be effective for this task, however, it has a substantial computing cost. We propose a constrained independent component analysis algorithm based on minimizing the entropy bound (c-EBM) to overcome the computational complexity limitation of IVA. A set of spatial maps used as constraints provides a connection across the datasets, provides alignment across subject-wise ICA analyses and serves as a foundation for subgroup identification. The approach makes use of the available prior knowledge while allowing flexible density modeling without an orthogonality requirement for the demixing matrix. Synthetic data and large scale multi-subject resting state fMRI data have both been used to evaluate the performance of the new algorithm, c-EBM. The findings demonstrate that c-EBM is adaptable in terms of various settings for the constraint parameter on the synthetic data. With multi-subject resting state fMRI data, c-EBM can effectively identify subgroups and discover meaningful brain networks that show significant group differences between subgroups.

BrainForge is a cloud-enabled, web-based analysis platform for neuroimaging research. This website allows users to archive data from a study and effortlessly process data on a high-performance computing cluster. After analyses are completed, results can be quickly shared with colleagues. BrainForge solves multiple problems for researchers who want to analyze neuroimaging data, including issues related to software, reproducibility, computational resources, and data sharing. BrainForge can currently process structural, functional, diffusion, and arterial spin labeling MRI modalities, including preprocessing and group level analyses. Additional pipelines are currently being added, and the pipelines can accept the BIDS format. Analyses are conducted completely inside of Singularity containers and utilize popular software packages including Nipype, Statistical Parametric Mapping, the Group ICA of fMRI Toolbox, and FreeSurfer. BrainForge also features several interfaces for group analysis, including a fully automated adaptive ICA approach.

Neurocognitive impairment continues to be common comorbidity for people living with HIV (PLWH). Given the chronic nature of HIV disease, identifying reliable biomarkers of these impairments is essential to advance our understanding of the underlying neural foundation and facilitate screening and diagnosis in clinical care. While neuroimaging provides immense potential for such biomarkers, to date, investigations in PLWH have been mostly limited to either univariate mass techniques or a single neuroimaging modality. In the present study, connectome-based predictive modeling (CPM) was proposed to predict individual differences of cognitive functioning in PLWH, using resting-state functional connectivity (FC), white matter structural connectivity (SC), and clinical relevant measures. We also adopted an efficient feature selection approach to identify the most predictive features, which achieved an optimal prediction accuracy of r = 0.61 in the discovery dataset (n = 102) and r = 0.45 in an independent validation HIV cohort (n = 88). Two brain templates and nine distinct prediction models were also tested for better modeling generalizability. Results show that combining multimodal FC and SC features enabled higher prediction accuracy of cognitive scores in PLWH, while adding clinical and demographic metrics may further improve the prediction by introducing complementary information, which may help better evaluate the individual-level cognitive performance in PLWH.

Feature selection can disclose biomarkers of mental disorders that have unclear biological mechanisms. Although neighborhood rough set (NRS) has been applied to discover important sparse features, it has hardly ever been utilized in neuroimaging-based biomarker identification, probably due to the inadequate feature evaluation metric and incomplete information provided under a single-granularity. Here, we propose a new NRS-based feature selection method and successfully identify brain functional connectivity biomarkers of schizophrenia (SZ) using functional magnetic resonance imaging (fMRI) data. Specifically, we develop a new weighted metric based on NRS combined with information entropy to evaluate the capacity of features in distinguishing different groups. Inspired by multi-granularity information maximization theory, we further take advantage of the complementary information from different neighborhood sizes via a multi-granularity fusion to obtain the most discriminative and stable features. For validation, we compare our method with six popular feature selection methods using three public omics datasets as well as resting-state fMRI data of 393 SZ patients and 429 healthy controls. Results show that our method obtained higher classification accuracies on both omics data (100.0%, 88.6%, and 72.2% for three omics datasets, respectively) and fMRI data (93.9% for main dataset, and 76.3% and 83.8% for two independent datasets, respectively). Moreover, our findings reveal biologically meaningful substrates of SZ, notably involving the connectivity between the thalamus and superior temporal gyrus as well as between the postcentral gyrus and calcarine gyrus. Taken together, we propose a new NRS-based feature selection method that shows the potential of exploring effective and sparse neuroimaging-based biomarkers of mental disorders.

Alzheimer's disease (AD) is a neurodegenerative process characterized by the accumulation of amyloid-beta plaques and neurofibrillary tangles and is the most common cause of dementia. Studies have been striving to analyze the disease using available physiological and behavioral data. Functional/structural neuroimaging and genomics are complementary modalities for exploring the mechanisms subserving the development of AD. In this paper, we present a deep multimodal generative data fusion framework for integrating these sources in a classification task involving AD patients and healthy controls from the ADNI database. Biological data fusion has the potential to improve the final prediction, but at the same time, it is particularly challenging due to the unavailability of all sources of input for the entire cohort of subjects. Our proposed model allows us to perform prediction even if individuals are missing certain modalities. Our method addresses the missing modalities problem via knowledge transfer from two generative adversarial networks. The model exhibits superior performance for predicting AD versus healthy control, even with missing modalities. This could have an important impact from the patient point of view since certain clinical tests may not be necessary or available to a given individual.

In recent years, many neuroimaging studies have begun to integrate gradient-based explainability methods to provide insight into key features. However, existing explainability approaches typically generate a point estimate of importance and do not provide insight into the degree of uncertainty associated with explanations. In this study, we present a novel approach for estimating explanation uncertainty for convolutional neural networks (CNN) trained on neuroimaging data. We train a CNN for classification of individuals with schizophrenia (SZs) and controls (HCs) using resting state functional magnetic resonance imaging (rs-fMRI) dynamic functional network connectivity (dFNC) data. We apply Monte Carlo batch normalization (MCBN) and generate an explanation following each iteration using layer-wise relevance propagation (LRP). We then examine whether the resulting distribution of explanations differs between SZs and HCs and examine the relationship between MCBN-based LRP explanations and regular LRP explanations. We find a number of significant differences in LRP relevance for SZs and HCs and find that traditional LRP values frequently diverge from the MCBN relevance distribution. This study provides a novel approach for obtaining insight into the level of uncertainty associated with gradient-based explanations in neuroimaging and represents a significant step towards increasing reliability of explainable deep learning methods within a clinical setting.

Schizophrenia (SZ) is a mental disorder that affects millions of people globally. At this time, diagnosis of SZ is based upon symptoms, which can vary from patient to patient and create difficulty with diagnosis. To address this issue, researchers have begun to look for neurological biomarkers of SZ and develop methods for automated diagnosis. In recent years, several studies have applied deep learning to raw EEG for automated SZ diagnosis. However, the use of raw time-series data makes explainability more difficult than it is for traditional machine learning algorithms trained on manually engineered features. As such, none of these studies have sought to explain their models, which is problematic within a healthcare context where explainability is a critical component. In this study, we apply perturbation-based explainability approaches to gain insight into the spectral and spatial features learned by two distinct deep learning models trained on raw EEG for SZ diagnosis for the first time. We develop convolutional neural network (CNN) and CNN long short-term memory network (CNN-LSTM) architectures. Results show that both models prioritize the T8 and C3 electrodes and the δ- and y-bands, which agrees with previous literature and supports the overall utility of our models. This study represents a step forward in the implementation of deep learning models for clinical SZ diagnosis, and it is our hope that it will inspire the more widespread application of explainability methods for insight into deep learning models trained for SZ diagnosis in the future.

Schizophrenia (SZ) is a neuropsychiatric disorder that adversely effects millions of individuals globally. Current diagnostic efforts are symptom based and hampered due to the variability in symptom presentation across individuals and overlap of symptoms with other neuropsychiatric disorders. This spawns the need for (1) biomarkers to aid with empirical SZ diagnosis and (2) the development of automated diagnostic approaches that will eventually serve in a clinical decision support role. In this study, we train random forest (RF) and support vector machine (SVM) models to differentiate between individuals with schizophrenia and healthy controls using spectral features extracted from resting state EEG data. We then perform two explainability analyses to gain insight into key frequency bands and channels. In our explainability analyses, we examine the reproducibility of SZ biomarkers across models with the goal of identifying those that have potential clinical implications. Our model performance results are well above chance level indicating the broader utility of spectral information for SZ diagnosis. Additionally, we find that the RF prioritizes the upper$\gamma$-band and is robust to loss of information from individual electrodes, while the SVM prioritizes the$\alpha$and$\theta$-bands and P4 and T8 electrodes. It is our hope that our findings will inform future efforts towards the empirical diagnosis of SZ and towards the development of clinical decision support systems for SZ diagnosis.

The application of clustering algorithms to fMRI functional network connectivity (FNC) data has been extensively studied over the past decade. When applied to FNC, these analyses assign samples to an optimal number of groups without a priori assumptions. Through these groupings, studies have provided insights into the dynamics of network connectivity through the identification of different brain states and have identified subgroups of individuals with unique brain activity. However, the manner in which underlying brain networks influence the identified groups is yet to be fully understood. In this study, we apply k-means clustering to resting-state fMRI-based static FNC data collected from 37,784 healthy individuals. We identified 2 groups of individuals with statistically significant differences in cognitive performance in several test metrics. Then, by applying two different versions of G2PC, a global permutation feature importance approach, and logistic regression with elastic net regularization, we were able to identify the relative importance of brain network pairs and their underlying features to the resulting groups. Through these approaches, together with the visualization of centroids' connectivity matrices, we were able to explain the observed differences in cognition in terms of specific key brain networks. We expect that our results will shed further light upon the effect of underlying brain networks on encountered cognitive differences between groups with unique brain activity.

Multimodal neuroimaging fusion studies of complex disorders like schizophrenia (SZ) have revealed disease-specific links between brain structure and function that unimodal analyses alone could not produce. Here, we utilize a multi-set CCA+ joint ICA fusion framework to study the connection between structural MRI and functional network connectivity (FNC) states derived via filter-banked connectivity (FBC), a unified method for estimating both static and dynamic FNC across the frequency spectrum. This joint analysis showed functional connections in both low-and high-frequency SZ-dominant states were significantly related to alterations in grey matter volume in several areas in the frontal and temporal cortices, which have formerly been implicated in SZ.

The decomposition of multi-subject fMRI data using rank-(L,L,1,1) block term decomposition (BTD) can preserve higher-way data structure and is more robust to noise effects by decomposing shared spatial maps (SMs) into a product of two rank-L loading matrices. However, since the number of whole-brain voxels is very large and rank L is larger than 1, the rank-(L,L,1,1) BTD requires high computation and memory. Therefore, we propose an accelerated rank-(L,L,1,1) BTD algorithm based upon the method of alternating least squares (ALS). We speed up updates of loading matrices by reducing fMRI data into subspaces, and add an orthonormality constraint on shared SMs to improve the performance. Moreover, we evaluate the rank-L effect on the proposed method for actual task-related fMRI data. The proposed method shows better performance when L=35. Meanwhile, experimental comparison results verify that the proposed method largely reduced (17.36 times) computation time compared to ALS while also providing satisfying separation performance.

Data fusion—the joint analysis of multiple datasets—through coupled factorizations has the promise to enable enhanced knowledge discovery, and hence is an active area. Various formulations of coupled matrix factorizations have been proposed, each with its own modeling assumptions. In this paper, we study two such methods, namely Independent Vector Analysis (IVA), i.e., extension of Independent Component Analysis (ICA) to multiple datasets, and PARAFAC2, a tensor factorization approach. We demonstrate the modeling assumptions of IVA and PARAFAC2 using simulations, revealing that both methods can accurately capture the latent components, albeit with certain differences in capturing the corresponding subject scores. By making use of a rich multi-task functional Magnetic Resonance Imaging (fMRI) dataset, we show how the two methods can be used for achieving two important goals at once, namely capturing group differences between patients with schizophrenia and healthy controls with interpretable components, as well as understanding the relationship across multiple tasks. This is achieved through the definition of source component vectors across datasets.

Identification of homogeneous subgroups of subjects plays a key role in the study of precision medicine. While there are a number of approaches based on the clustering of low-level features such as behavioral variables, work that makes use of fully multivariate nature of medical imaging data is very limited. Given that the individual variability in brain functional networks obtained from functional magnetic resonance imaging (fMRI) data is noted as being both significant and consistent like fingerprints, its use provides a particularly appealing approach to this challenging problem. We present a completely data-driven approach, subgroup identification using independent vector analysis (SI-IVA), which leverages the desirable properties of IVA to uncover the relationship across subjects along with the discovery of subgroup structures revealed by Gershgorin disc theorem. We show that SI-IVA outperforms an eigenanalysis-based approach by simulations. We then apply the method to real fMRI data obtained from patients of during resting state to identify group differences in multiple relevant brain regions including primary somatosensory and motor cortex, which demonstrates that SI-IVA provides interpretable and meaningful results.

Artificial intelligence has become pervasive across disciplines and fields, and biomedical image and signal processing is no exception. The growing and widespread interest on the topic has triggered a vast research activity that is reflected in an exponential research effort. Through study of massive and diverse biomedical data, machine and deep learning models have revolutionized various tasks such as modeling, segmentation, registration, classification and synthesis, outperforming traditional techniques. However, the difficulty in translating the results into biologically/clinically interpretable information is preventing their full exploitation in the field. Explainable AI (XAI) attempts to fill this translational gap by providing means to make the models interpretable and providing explanations. Different solutions have been proposed so far and are gaining increasing interest from the community. This paper aims at providing an overview on XAI in biomedical data processing and points to an upcoming Special Issue on Deep Learning in Biomedical Image and Signal Processing of the IEEE Signal Processing Magazine that is going to appear in March 2022.

Schizophrenia (SZ) is a serious psychiatric disorder, causing substantial socioeconomic burden. Since an SZ patients’ brain may have structural changes including reduced hippocampal and thalamic volume [1], brain MRI is becoming a popular imaging method studying SZ. In addition, as a genetic disorder, genetic information such as single nucleotide polymorphisms (SNP) plays an important role in distinguishing SZ. However, the structural and genetic changes in SZ patients are too subtle to be identified by human vision, so it is necessary to develop an automated method to find the nonlinear patterns associated with disease progression. Toward this, we propose a novel multi-modal deep learning approach where we combine both features from structural MRI (sMRI) and single-nucleotide polymorphisms (SNPs) for SZ classification. For sMRI, we extract convolutional features from a pre-trained deep neural network to capture morphological characteristics. For SNPs, we apply a layer-wise relevance propagation (LRP) method on a pre-trained 1-D convolutional network to identify SZ-linked SNPs. We then feed the combined features to a tree-based classifier for SZ diagnosis. Experimental results on clinical dataset showed classification accuracy was increased by 5.3% compared to the state of the art DenseNet using only sMRI data.

Recently, methods that represent data as a graph, such as graph neural networks (GNNs) have been successfully used to learn data representations and structures to solve classification and link prediction problems. The applications of such methods are vast and diverse, but most of the current work relies on the assumption of a static graph. This assumption does not hold for many highly dynamic systems, where the underlying connectivity structure is non-stationary and is mostly unobserved. Using a static model in these situations may result in suboptimal performance. In contrast, modeling changes in graph structure with time can provide information about the system whose applications go beyond classification. Most work of this type does not learn effective connectivity and focuses on cross-correlation between nodes to generate undirected graphs. An undirected graph is unable to capture direction of an interaction which is vital in many fields, including neuroscience. To bridge this gap, we developed dynamic effective connectivity estimation via neural network training (DECENNT), a novel model to learn an interpretable directed and dynamic graph induced by the downstream classification/prediction task. DECENNT outperforms state-of-the-art (SOTA) methods on five different tasks and infers interpretable task-specific dynamic graphs. The dynamic graphs inferred from functional neuroimaging data align well with the existing literature and provide additional information. Additionally, the temporal attention module of DECENNT identifies time-intervals crucial for predictive downstream task from multivariate time series data.

Privacy protection is one of the most crucial factors when sharing MR images between researchers. There are many defacing software programs that can blur or remove the face of MR images. However, there are reasons to believe that the brain and other remaining features are not only identifiable but also can be used for facial reconstruction to fill the face of the subject back into the image, and it is possible to rebuild the facial part of the images using recently developed machine learning models. A demonstration of this is of practical significance as it could convince the community to adopt stricter data-sharing standards. Additionally, even if the reconstructed faces are not identifiable, smooth completion of the “damaged” MRI images may improve methods that depend on the head modeling, such as source localization approaches. Recent work has been focusing on using the generative adversarial networks (GAN) for this task, which is generally believed to be the best method. Because we hope the model can generate the face entirely depending on the information from the brain, we show here an alternative approach, pixel constrained CNN, which is a purely supervised facial reconstruction. We simulated the rebuild process and showed convincing results.

Convolutional neural networks (CNNs) have shown promising results in classifying individuals with mental disorders such as schizophrenia using resting-state fMRI data. However, complex-valued fMRI data is rarely used since additional phase data introduces high-level noise though it is potentially useful information for the context of classification. As such, we propose to use spatial source phase (SSP) maps derived from complex-valued fMRI data as the CNN input. The SSP maps are not only less noisy, but also more sensitive to spatial activation changes caused by mental disorders than magnitude maps. We build a 3D-CNN framework with two convolutional layers (named SSPNet) to fully explore the 3D structure and voxel-level relationships from the SSP maps. Two interpretability modules, consisting of saliency map generation and gradient-weighted class activation mapping (Grad-CAM), are incorporated into the well-trained SSPNet to provide additional information helpful for understanding the output. Experimental results from classifying schizophrenia patients (SZs) and healthy controls (HCs) show that the proposed SSPNet significantly improved accuracy and AUC compared to CNN using magnitude maps extracted from either magnitude-only (by 23.4 and 23.6% for DMN) or complex-valued fMRI data (by 10.6 and 5.8% for DMN). SSPNet captured more prominent HC-SZ differences in saliency maps, and Grad-CAM localized all contributing brain regions with opposite strengths for HCs and SZs within SSP maps. These results indicate the potential of SSPNet as a sensitive tool that may be useful for the development of brain-based biomarkers of mental disorders.

With the development of multi-model neuroimaging technology and gene detection technology, the efforts of integrating multi-model imaging genetics data to explore the virulence factors of schizophrenia (SZ) are still limited. To address this issue, we propose a novel algorithm called group sparse of joint non-negative matrix factorization on orthogonal subspace (GJNMFO). Our algorithm fuses single nucleotide polymorphism (SNP) data, function magnetic resonance imaging (fMRI) data and epigenetic factors (DNA methylation) by projecting three-model data into a common basis matrix and three different coefficient matrices to identify risk genes, epigenetic factors and abnormal brain regions associated with SZ. Specifically, we introduce orthogonal constraints on the basis matrix to discard unimportant features in the row of coefficient matrices. Since imaging genetics data have rich group information, we draw into group sparse on three coefficient matrices to make the extracted features more accurate. Both the simulated and real Mind Clinical Imaging Consortium (MCIC) datasets are performed to validate our approach. Simulation results show that our algorithm works better than other competing methods. Through the experiments of MCIC datasets, GJNMFO reveals a set of risk genes, epigenetic factors and abnormal brain functional regions, which have been verified to be both statistically and biologically significant.

Tucker decomposition can provide an intuitive summary to understand brain function by decomposing multi-subject fMRI data into a core tensor and multiple factor matrices, and was mostly used to extract functional connectivity patterns across time/subjects using orthogonality constraints. However, these algorithms are unsuitable for extracting common spatial and temporal patterns across subjects due to distinct characteristics such as high-level noise. Motivated by a successful application of Tucker decomposition to image denoising and the intrinsic sparsity of spatial activations in fMRI, we propose a low-rank Tucker-2 model with spatial sparsity constraint to analyze multi-subject fMRI data. More precisely, we propose to impose a sparsity constraint on spatial maps by using an$ \ell _{p} $norm (${0}< {p}\le {1}$), in addition to adding low-rank constraints on factor matrices via the Frobenius norm. We solve the constrained Tucker-2 model using alternating direction method of multipliers, and propose to update both sparsity and low-rank constrained spatial maps using half quadratic splitting. Moreover, we extract new spatial and temporal features in addition to subject-specific intensities from the core tensor, and use these features to classify multiple subjects. The results from both simulated and experimental fMRI data verify the improvement of the proposed method, compared with four related algorithms including robust Kronecker component analysis, Tucker decomposition with orthogonality constraints, canonical polyadic decomposition, and block term decomposition in extracting common spatial and temporal components across subjects. The spatial and temporal features extracted from the core tensor show promise for characterizing subjects within the same group of patients or healthy controls as well.

Recent studies show that multi-modal data fusion techniques combine information from diverse sources for comprehensive diagnosis and prognosis of complex brain disorder, often resulting in improved accuracy compared to single-modality approaches. However, many existing data fusion methods extract features from homogeneous networs, ignoring heterogeneous structural information among multiple modalities. To this end, we propose a Hypergraph-based Multi-modal data Fusion algorithm, namely HMF. Specifically, we first generate a hypergraph similarity matrix to represent the high-order relationships among subjects, and then enforce the regularization term based upon both the inter- and intra-modality relationships of the subjects. Finally, we apply HMF to integrate imaging and genetics datasets. Validation of the proposed method is performed on both synthetic data and real samples from schizophrenia study. Results show that our algorithm outperforms several competing methods, and reveals significant interactions among risk genes, environmental factors and abnormal brain regions.

Neuroscience studies have begun to benefit from combining large numbers of data from different sites to increase statistical power. Pooling data from various sites into a single analysis introduces additional variability from site-effects due to differences in scanner protocols, imaging protocol, and acquisition methods, among others. These site-effects can reduce statistical power or lead to erroneous conclusions. Harmonization is the process of combining data aiming at reducing site variability. One recent approach for harmonizing data called ComBat has been shown to be helpful in the context of functional MRI and static functional connectivity. However, ComBat has not been applied to the analysis of dynamic functional network connectivity (dFNC). Here we explore the impact of ComBat harmonization on dFNC data collected from two different mild traumatic brain injury (mTBI) studies. Results show that ComBat harmonization of dFNC can reduce site effects producing a more robust analysis of patient effects across sites.

The reliability of functional network connectivity (FNC) measured using independent component analysis (ICA) has frequently been explored within the literature, with results displaying varying levels of reliability and demonstrating that minor changes in data preprocessing procedures can significantly alter FC results and reliability. However, one important avenue of research that has not been explored within the current literature is the effect of spatial normalization techniques on FNC reliability. Spatially constrained independent component analysis techniques such as multi-objective optimization with reference (MOO-ICAR) is one of many methods used to study brain functional connectivity (FC) using fMRI that is theoretically robust to variations which may arise in data as a result of normalization procedures. In this work, we deploy MOO-ICAR across 30 different spatial normalization pipelines varying across participant template, normalization modality (anatomical vs functional), and one vs. two-stage warps to MNI space. Most components display relatively high consistency intraclass-correlation coefficients (ICCs), with the vast majoritv (~80%) ereater than 0.5.

Spectral analysis of electroencephalography (EEG) data has developed as an important area of EEG research. EEG spectra have been analyzed with explainable machine learning and deep learning methods. However, as deep learning has developed, many studies have used raw EEG data, which is poorly suited for traditional explainability methods. Several studies have introduced methods for spectral insight into classifiers trained on raw EEG data. These studies have provided global insight into the frequency bands important to a classifier but not local insight into the frequency bands important to the classification of individual samples. Local explainability could be particularly helpful for EEG domains like sleep stage classification that feature multiple evolving states. We present a novel local spectral explainability approach and use it to explain a convolutional neural network trained for automated sleep stage classification. We use our approach to show how the importance of frequency bands varies over time and even within the same sleep stages. Also, to better understand how our approach compares to existing methods, we compare a global estimate of spectral importance generated from our local results with an existing global spectral importance approach. We find that the δ band is most important for most sleep stages, though ß is most important for the non-rapid eye movement 2 (NREM2) sleep stage. Additionally, 0 is particularly important for identifying Awake and NREM1 samples. Our study represents the first approach developed for local spectral insight into deep learning classifiers trained on raw EEG time series.

In recent years, more biomedical studies have begun to use multimodal data to improve model performance. Many studies have used ablation for explainability, which requires the modification of input data. This can create out-of-distribution samples and lead to incorrect explanations. To avoid this problem, we propose using a gradient-based feature attribution approach, called layer-wise relevance propagation (LRP), to explain the importance of modalities both locally and globally for the first time. We demonstrate the feasibility of the approach with sleep stage classification as our use-case and train a 1-D convolutional neural network with electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG) data. We also analyze the relationship of our local explainability results with clinical and demographic variables to determine whether they affect our classifier. Across all samples, EEG is the most important modality, followed by EOG and EMG. For individual sleep stages, EEG and EOG have higher relevance for awake and non-rapid eye movement 1 (NREM1). EOG is most important for REM, and EEG is most relevant for NREM2-NREM3. Also, LRP gives consistent levels of importance to each modality for the correctly classified samples across folds but inconsistent levels of importance for incorrectly classified samples. Our statistical analyses suggest that medication has a significant effect upon patterns learned for EEG and EOG NREM2 and that subject sex and age significantly affects the EEG and EOG patterns learned, respectively. Our results demonstrate the viability of gradient-based approaches for explaining multimodal electrophysiology classifiers and suggest their generalizability for other multimodal classification domains.

With the growing use of multimodal data for deep learning classification in healthcare research, more studies are presenting explainability methods for insight into multimodal classifiers. Among these studies, few utilize local explainability methods, which can provide (1) insight into the classification of samples over time and (2) better understanding of the effects of demographic and clinical variables upon patterns learned by classifiers. To the best of our knowledge, we present the first local explainability approach for insight into the importance of each modality to the classification of samples over time. Our approach uses ablation, and we demonstrate how it can show the importance of each modality to the correct classification of each class. We further present a novel analysis that explores the effects of demographic and clinical variables upon the multimodal patterns learned by the classifier. As a use-case, we train a convolutional neural network for automated sleep staging with electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG) data. We find that EEG is the most important modality across most stages, though EOG is particularly important for non-rapid eye movement stage 1. Further, we identify significant relationships between the local explanations and subject age, sex, and state of medication which suggest that the classifier learned features associated with these variables across multiple modalities and correctly classified samples. Our novel explainability approach has implications for many fields involving multimodal classification. Moreover, our examination of the degree to which demographic and clinical variables may affect classifiers could provide direction for future studies in automated biomarker discovery.

Diagnosis of bipolar disorder (BD) patients without evident mania can lead to misdiagnosis as major depressive disorder (MDD). The current Diagnostic and Statistical Manual (DSM) is not based on pathopsychology, and the diagnosis based on DSM can lead to an imperfect prediction of medication-class of response in such complex cases. Clinicians using the DSM can spend months or even years choosing a medication for a patient using a process of trial and error. To improve this situation, a biologically based classification algorithm is needed. Osuch et al. (2018) presented a kernel support vector machine (SVM)-based algorithm to predict the medication-class of response from new patient samples whose diagnoses were unclear. Here we extend their work by applying a robust, fully automated Neu-romark independent component analysis framework to extract comparable features in a multi-dataset setting and learning a kernel function for SVM based on different subspaces from multiple modalities. The Neuromark framework was successful in replicating the prior result with 95.45% accuracy (sensitivity 90.24%, specificity 92.3%). We further incorporated two additional datasets comprising bipolar disorder (BD) and major depressive disorder (MDD) patients. We validated the trained algorithm on these datasets, resulting in a testing accuracy of up to 87.48% (sensitivity 95%, specificity 91.36%) without using any site or scanner harmonization techniques. This approach can help reveal biological markers of medication-class of response within mood disorders in clinical settings.

Spectral analysis remains a hallmark approach for gaining insight into electrophysiology modalities like electroencephalography (EEG). As the field of deep learning has progressed, more studies have begun to train deep learning classifiers on raw EEG data, which presents unique problems for explainability. A growing number of studies have presented explainability approaches that provide insight into the spectral features learned by deep learning classifiers. However, existing approaches only attribute importance to different frequency bands. Most of the methods cannot provide insight into the actual spectral values or the relationship between spectral features that models have learned. Here, we present a novel adaptation of activation maximization for electrophysiology time-series that generates samples that indicate the features learned by classifiers by optimizing their spectral content. We evaluate our approach within the context of EEG sleep stage classification with a convolutional neural network, and we find that our approach is able to identify spectral patterns known to be associated with each sleep stage. We also find surprising results suggesting that our classifier may have prioritized the use of eye and motion artifact when identifying Awake samples. Our approach is the first adaptation of activation maximization to the domain of raw electrophysiology classification. Additionally, our approach has implications for explaining any classifier trained on highly dynamic, long time-series.

The general psychopathology factor (p factor), derived from a wide range of psychological symptoms, is proposed to approximate an individual’s tendency to develop a broad range of psychiatric disorders. The aim of this study was to extract the general p factor in a bifactor model from confirmatory factor analysis (CFA), as well as internalizing and externalizing factors, and test the stability of the factors derived from different behavioral measures. Using data from the Adolescent Brain and Cognitive Development (ABCD) study, we compared the latent factors derived from Child Behavior Checklist measure (a standard approach) with those from broader measures. Multiple linear regression was used to assess the relationship of the p factor with other behavioral or cognition measures and clinical diagnoses. The results showed that the p factor had greater stability and a higher correlation between models (r=0.87) than internalizing and externalizing factors (r=0.32,0.02, respectively). The p factor explained significant variances in all dimensional behavioral variables as well as neurocognition and was significantly related to all mental disorders available. A mixed-effects model was constructed to measure the association of the p factor with screen time activity, sleep duration, physical activity, household income, and gender (altogether, these explained 8.02% of the variance in the p factor). The findings from this study also show that the general p factor defined in the bifactor model has sufficient determinacy.

This work uses a variational autoencoder (VAE) to perform non-linear representation learning from voxelwise rs-fMRI data. The VAE learns a non-linear dimensionality reduction of the data in the form of a latent vector. These latent vectors retain meaningful information related to a subject’s demographics and clinical diagnosis. The retention of meaningful information in the latent vectors is evaluated using age regression and sex classification tasks on the UK Biobank dataset. The results on these tasks are highly encouraging and a linear regressor trained on the latent vectors to predict age performs almost on par with a supervised neural network. Further, the same latent vectors can almost perfectly linearly separate sex. The model that is pre-trained on UK Biobank is also fine-tuned on a smaller neuropsychiatric dataset for a varying number of epochs. The latent vectors it generates for this dataset are then evaluated by performing a schizophrenia diagnosis classification task. We find that pre-training the model on UK Biobank significantly improves the quality of the latent vectors and that the vectors themselves are fairly discriminative. To understand the structure of the latent vectors with respect to demographic variables or neuropsychiatric disorders we train a variety of supervised models on the latent vectors. The results presented in this work open up more in-depth research into the factors of variation that the VAE models and how they can be improved for voxelwise rs-fMRI data.

Anxiety and Depression are currently among the most common mental disorders in children and adolescents. Both genetics and environments play an important role in the development and progress of disorders. This study aimed to understand the effect of multiscale environmental factors on anxiety and depression in school-age children, to refine the identification of genetic variants contributing to susceptibility to anxiety and depression, and to evaluate the genetic heritability. We analyzed data from the Adolescent Brain and Cognitive Development study and computed one principal factor to present the overall anxiety and depression scale in 11,875 participants with ages between 9 and 10 years old. Linear mixed-effect models along with the recursive feature elimination regression and LASSO regression models were used to determine the environmental effects from the macro scale (population density, air pollution), meso scale (neighborhood, school), to micro scale (family and individual experience). Genome-wide association analyses were then performed controlling for environmental factors and sample relatedness to determine the susceptible genetic variants. Furthermore, heritability was calculated for the white population. The results showed that six environmental factors (early life stress, household income, population density, area crime, neighborhood safety, and school risk) and sex had significant effect on anxiety/depression score. Genome-wide association tests showed no SNPs reached a genome-wide significance (p=5e-08), but some genetic mutations including SNPs in SCN1A showed promising effects, and the heritability was estimated close to 15% in the white population.

Tucker decomposition (TKD) has been utilized to identify functional connectivity patterns using processed fMRI data, but seldom focuses on originally acquired fMRI data. This study proposes to decompose multi-subject fMRI data in a natural three-way of voxel × time × subject via TKD. Different from existing tensor decomposition algorithms such as canonical polyadic decomposition (CPD) for extracting shared spatial maps (SMs), we propose to extract both shared and individual SMs by exploring spatial-temporal-subject relationship contained in the core tensor. We test the proposed method using multi-subject resting-state fMRI data with comparison to CPD for evaluating shared SMs and independent vector analysis (IVA) for assessing individual SMs under different model orders. The results show that the proposed method yields better and more robust shared SMs than CPD and more consistent individual SMs than IVA, indicating the potential of TKD in providing group and individual brain networks in a high-dimensional coupling way.

Spatial source phase (SSP), derived from complex-valued functional magnetic resonance imaging (fMRI) data by data-driven methods, has unique capacity of identifying blood oxygenation-level dependent (BOLD)-related voxels from noisy voxels regardless of their amplitudes. However, the use of SSP constraint in sparse representation algorithms have rarely been studied. This study proposes a sparse representation method using SSP hard thresholding to achieve the sparsity of spatial components, enabling the use of initially complex-valued fMRI data and retaining the brain information embedded in noisy voxels and weak BOLD-related voxels with small phase values. Rank-1 matrix estimation is applied to sequentially update dictionary atoms and corresponding spatial components, followed by hard thresholding on spatial components based on SSP. The proposed method is evaluated using both simulated and experimental complex-valued data. The results show that the proposed method yields better performance than a complex-valued dictionary learning algorithm when using initially acquired complex-valued task-related fMRI data.

Convolutional neural network (CNN) can achieve better classification by using independent component analysis (ICA) components derived from complex-valued fMRI data than from magnitude-only fMRI data due to incorporating additional phase information. However, thus far magnitude slices of only a single brain network (i.e. spatial component) has been used in the classification. This study aims to take advantages of multiple ICA components in providing rich information and to provide a conclusion for efficient multiple-component fusion. More precisely, we present three fusion approaches: 1) averaging multiple ICA components as inputs of a single-component CNN, 2) concatenating features of multiple single-component CNN, and 3) averaging predictive probabilities of multiple single-component CNN. We evaluate the proposed methods using resting-state fMRI data collected from 42 schizophrenia patients and 40 healthy controls. Experimental results show that all three fusion approaches can improve classification accuracy compared to the single-component CNN, and the first approach performs the best. No matter which fusion method is used, we reach the same conclusion that four-component fusion is sufficient to obtain satisfying performance, and two-component fusion yields higher improvement and better performance than the single-component classification, especially when using components having good accuracy for single-component CNN classification.

Recent advances in imaging genetics make it possible to combine different types of data including medical images like functional magnetic resonance imaging (fMRI) and genetic data like single nucleotide polymorphisms (SNPs) for comprehensive diagnosis of mental disorders. Understanding complex interactions among these heterogeneous data may give rise to a new perspective, while at the same time demand statistical models for their integration. Various graphical models have been proposed for the study of interaction or association networks with continuous, binary, and count data as well as the mixture of them. However, limited efforts have been made for the multinomial case, for instance, SNP data. Our goal is therefore to fill the void by developing a graphical model for the integration of fMRI image and SNP data, which can provide deeper understanding of the unknown neurogenetic mechanism. In this article, we propose a latent Gaussian copula model for mixed data containing multinomial components. We assume that the discrete variable is obtained by discretizing a latent (unobserved) continuous variable and then create a semi-rank based estimator of the graph structure. The simulation results demonstrate that the proposed latent correlation has more steady and accurate performance than several existing methods in detecting graph structure. When applying to a real schizophrenia data consisting of SNP array and fMRI image collected by the Mind Clinical Imaging Consortium (MCIC), the proposed method reveals a set of distinct SNP-brain associations, which are verified to be biologically significant. The proposed model is statistically promising in handling mixed types of data including multinomial components, which can find widespread applications. To promote reproducible research, the R code is available at https://github.com/Aiying0512/LGCM.

Unsupervised latent variable models-blind source separation (BSS) especially-enjoy a strong reputation for their interpretability. But they seldom combine the rich diversity of information available in multiple datasets, even though multidatasets yield insightful joint solutions otherwise unavailable in isolation. We present a direct, principled approach to multidataset combination that takes advantage of multidimensional subspace structures. In turn, we extend BSS models to capture the underlying modes of shared and unique variability across and within datasets. Our approach leverages joint information from heterogeneous datasets in a flexible and synergistic fashion. We call this method multidataset independent subspace analysis (MISA). Methodological innovations exploiting the Kotz distribution for subspace modeling, in conjunction with a novel combinatorial optimization for evasion of local minima, enable MISA to produce a robust generalization of independent component analysis (ICA), independent vector analysis (IVA), and independent subspace analysis (ISA) in a single unified model. We highlight the utility of MISA for multimodal information fusion, including sample-poor regimes ( N = 600 ) and low signal-to-noise ratio, promoting novel applications in both unimodal and multimodal brain imaging data.

Functional magnetic resonance imaging (fMRI) is a powerful technique with the potential to estimate individual variations in behavioral and cognitive traits. Joint learning of multiple datasets can utilize their complementary information so as to improve learning performance, but it also gives rise to the challenge for data fusion to effectively integrate brain patterns elicited by multiple fMRI data. However, most of the current data fusion methods analyze each single dataset separately and further infer the relationship among them, which fail to utilize the multidimensional structure inherent across modalities and may ignore complex but important interactions. To address this issue, we propose a novel sparse tensor decomposition method to integrate multiple task-stimulus (paradigm) fMRI data. Seeing each paradigm fMRI as one modality, our proposed method considers the relationships across subjects and modalities simultaneously. In specific, a third-order tensor is first modeled by using the functional network connectivity (FNC) of subjects in multiple fMRI paradigms. A novel sparse tensor decomposition with the regularization terms is designed to factorize the tensor into a series of rank-one components, which can extract the shared components across modalities as the embedded features. The L2,1-norm regularizer (i.e., group sparsity) is enforced to select a few common features among multiple subjects. Validation of the proposed method is performed on realistic three paradigm fMRI datasets from the Philadelphia Neurodevelopmental Cohort (PNC) study, for the study of the relationship between the FNC and human cognitive abilities. Experimental results show our method outperforms several other competing methods in the prediction of individuals with different cognitive behaviors via the wide range achievement test (WRAT). Furthermore, our method discovers the FNC related to the cognitive behaviors, such as the connectivity associated with the default mode network (DMN) for three paradigms, and the connectivity between DMN and visual (VIS) domains within the emotion task.

With the development of neuroimaging techniques, a growing amount of multi-modal brain imaging data are collected, facilitating comprehensive study of the brain. In this paper, we jointly analyzed functional magnetic resonance imaging (fMRI) collected under different paradigms in order to understand cognitive behaviors of an individual. To this end, we proposed a novel multi-view learning algorithm called structure-enforced collaborative regression (SCoRe) to extract co-expressed discriminative brain regions under the guidance of anatomical structure of the brain. An advantage of SCoRe over its predecessor collaborative regression (CoRe) lies in its incorporation of group structures in the brain imaging data, which makes the model biologically more meaningful. Results from real data analysis has confirmed that by incorporating prior knowledge of brain structure, SCoRe can deliver better prediction performance and is less sensitive to hyper-parameters than CoRe. After validation with simulation experiments, we applied SCoRe to fMRI data collected from the Philadelphia Neurodevelopmental Cohort and adopted the scores from the wide range achievement test (WRAT) to evaluate an individual's cognitive skills. We located 14 relevant brain regions that can efficiently predict WRAT scores and these brain regions were further confirmed by other independent studies.

The combination of multimodal imaging and genomics provides a more comprehensive way for the study of mental illnesses and brain functions. Deep network-based data fusion models have been developed to capture their complex associations, resulting in improved diagnosis of diseases. However, deep learning models are often difficult to interpret, bringing about challenges for uncovering biological mechanisms using these models. In this work, we develop an interpretable multimodal fusion model to perform automated diagnosis and result interpretation simultaneously. We name it Grad-CAM guided convolutional collaborative learning (gCAM-CCL), which is achieved by combining intermediate feature maps with gradient-based weights. The gCAM-CCL model can generate interpretable activation maps to quantify pixel-level contributions of the input features. Moreover, the estimated activation maps are class-specific, which can therefore facilitate the identification of biomarkers underlying different groups. We validate the gCAM-CCL model on a brain imaging-genetic study, and demonstrate its applications to both the classification of cognitive function groups and the discovery of underlying biological mechanisms. Specifically, our analysis results suggest that during task-fMRI scans, several object recognition related regions of interests (ROIs) are activated followed by several downstream encoding ROIs. In addition, the high cognitive group may have stronger neurotransmission signaling while the low cognitive group may have problems in brain/neuron development due to genetic variations.

Understanding the associations of the structural and functional patterns of the brain is vital. Recent studies have focused on utilizing this information within and across the different functional and anatomical domains (i.e., groups of brain networks) using neuroimaging data. In this work, we use a Bayesian optimization-based method known as the Tree Parzen Estimator (TPE) to identify variation in the nature of information encoded by different functional magnetic resonance imaging (fMRI) sub-domains of the brain. We show by repeated cross-validation on a schizophrenia classification task that specific sub-domains may require more sophisticated learning architectures to contribute optimally to classification, while others require less complicated ones. Our findings reveal the need for adaptive, hierarchical learning frameworks catering to features from different sub-domains to optimally identify features enabling the prediction of the outcome of interest.

Objective: A graph theoretical approach provides a powerful framework for discovering potential biomarkers of psychotic disorders. Comparing the brain graphs of the control and patient groups can help us to discover changes in mental disorders in a more convenient way. In this paper, we propose a novel tool to identify missing links associated with blocked paths (segregation) and new links associated with additional paths (abnormal integration) in estimated patient group's time-varying graphs. We highlight the approach in an example application to the resting-state functional magnetic resonance imaging (fMRI) data of schizophrenia (SZ) patients. Methods: We first estimated whole-brain functional connectivity dynamics using a combination of spatial independent component analysis (ICA), sliding time window, and k-means clustering of windowed correlation matrices on resting-state fMRI data. The clusters are regarded as functional connectivity states, and each of them includes time intervals exhibiting similar connectivity patterns. We then estimated a Gaussian graphical model (GGM) for each state for both groups. To evaluate this approach, we compared different paths between brain components (nodes) of SZ and control groups' graphs within each state by using the concept of connected components in graph theory. Results: We identified missing edges associated with disconnectivity (disconnectors) and showed there are additional edges in the SZ group graph that contribute to creating new paths in brain graphs. Conclusion: The proposed approach provides a tool for extracting time-varying graphs and identifying disconnectors associated with disconnectivity (absence of paths) and also connectors associated with abnormal integration (additional paths) in patient group graphs. Significance: We detected several missing links in SZ, both within and between functional domains, in particular within the subcortical (3 links) and somatomotor (4 links) domains. Interestingly, our proposed method identified new links within the somatomotor domain which may be related to a compensatory response in patients that warrants future study.

Resting-state brain networks (RSNs) have been recently used extensively for different tasks such as age prediction and patient classification. These networks are identified through analysis of synchronous low-frequency fluctuations in blood oxygenation level dependent (BOLD) signals obtained from resting-state functional magnetic resonance imaging (fMRI) scans. A significant majority of studies published so far involve time series analysis of fMRI signals to identify their endpoint of interest and less research has analyzed the connection between spatial networks of the brain to discover novel biomarkers. In this study, we show, for the first time, that the interaction between RSNs can uniquely characterize individual subjects. We propose a novel approach called BrainNet, a deep Siamese-based 3D convolutional neural network that learns to compare subjects by capturing the interactions between brain networks. We show that, our trained model can accurately discriminate subjects using pairs of networks as input and that it generalizes to the unseen cases. We also show the proposed model captures age-and gender-rich features in characterizing patterns of network-network interactions notwithstanding any supervision.

Brain parcellation plays a significant role in computational neuroimaging by dividing the brain into meaningful anatomical sub-regions which can be used to study broad brain functionalities and structures. Most ongoing brain parcellation research has focused on fixed regions that do not vary across individuals or over time. However, brain functional organization is by its very nature dynamic and ignoring this can lead to misleading results. In this work, we have tried to address this shortcoming in fMRI-based brain parcellation by proposing a novel 4D approach using a deep residual network structure, trained to predict probabilities of voxels in each volume belonging to independent components extracted from fMRI images. Results show that the presented approach not only provides informative 4D spatiotemporal networks which are individualized but also linked across subjects, providing an important tool for further study of the human brain.

In this study, we classified older (63 years old) from middle adult (45-63 years old) subjects by estimating whole-brain functional network connectivity (FNC) including the connectivity among subcortical network (SCN), auditory network (ADN), sensorimotor network (SMN), visual sensory network (VSN), cognitive control network (CCN), default mode network (DMN), cerebellar network (CBN) from the adult subjects (n = 9394; 45-81 y). We used three tree-based classifiers, including random forest (RF), XGBoost, and CATBoost. Next, we leveraged the SHapley Additive exPlanations (SHAP) approach as an explainable feature learning method to model the difference between the brain connectivity of the old and middle adult subjects. Opposed to the conventional statistical learning, which typically assesses each feature separately, the explainable machine learning method used here offers a generalized model in the connectivity difference between older and middle adults. Based on this method, we found that all three models successfully differentiate middle adult adults from older adults based on wholebrain FNC. We also found that all brain networks contributed to the top 20 features selected by the SHAP method in all three models. We highlighted the role of the CCN and SNC in differentiating between these two groups.

Functional magnetic resonance imaging (fMRI) is a brain imaging technique which provides detailed insights into brain function and its disruption in various brain disorders. fMRI data can be analyzed using data-driven or region-of-interest based methods. The data-driven analysis of brain activity maps involves several steps, the first of which is identifying whether the maps capture what might be interpreted as intrinsic connectivity networks (ICNs) or artifacts. This is followed by linking the ICNs to known anatomical and/or functional parcellations. Optionally, as in the study of functional network connectivity (FNC), rearranging the connectivity graph is also necessary for systematic interpretation. Here we present a toolbox that automates all these processes under minimal or no supervision with high accuracy. We provide a pretrained cross-validated elastic-net regularized general linear model for the noisecloud toolbox to separate the ICNs from artifacts. We include several well-known anatomical and functional parcellations from which researchers can choose to label the activity maps. Finally, we integrate a method for maximizing the within-domain modularity to generate a more systematically structured FNC matrix. We improve upon and integrate existing techniques and new methods to design this toolbox which can take care of all the above needs. Specifically, we show that our pretrained model achieves 89% accuracy and 100% precision at classifying ICNs from artifacts in a validation dataset. Researchers are generating brain imaging data and analyzing brain activity at an ever-increasing rate. The Autolabeller toolbox can help automate such analyses for faster and reproducible research.

Characterizing time-evolving networks is a challenging task, but it is crucial for understanding the dynamic behavior of complex systems such as the brain. For instance, how spatial networks of functional connectivity in the brain evolve during a task is not well-understood. A traditional approach in neuroimaging data analysis is to make simplifications through the assumption of static spatial networks. In this paper, without assuming static networks in time and/or space, we arrange the temporal data as a higher-order tensor and use a tensor fac-torization model called PARAFAC2 to capture underlying patterns (spatial networks) in time-evolving data and their evolution. Numerical experiments on simulated data demonstrate that PARAFAC2 can successfully reveal the underlying networks and their dynamics. We also show the promising performance of the model in terms of tracing the evolution of task-related functional connectivity in the brain through the analysis of functional magnetic resonance imaging data.

Schizophrenia (SZ) is a complex disease. Single nucleotide polymorphism (SNP), brain activity measured by functional magnetic resonance imaging (fMRI) and DNA methylation are all important biomarkers that can be used for the study of SZ. To our knowledge, there has been little effort to combine these three datasets together. In this study, we propose a group sparse joint nonnegative matrix factorization (GSJNMF) model to integrate SNP, fMRI, and DNA methylation for the identification of multi-dimensional modules associated with SZ, which can be used to study regulatory mechanisms underlying SZ at multiple levels. The proposed GSJNMF model projects multiple types of data onto a common feature space, in which heterogeneous variables with large coefficients on the same projected bases are used to identify multi-dimensional modules. We also incorporate group structure information available from each dataset. The genomic factors in such modules have significant correlations or functional associations with several brain activities. At the end, we have applied the method to the analysis of real data collected from the Mind Clinical Imaging Consortium (MCIC) for the study of SZ and identified significant biomarkers. These biomarkers were further used to discover genes and corresponding brain regions, which were confirmed to be significantly associated with SZ.

The papers in this special section examine important current topics on multimodal data fusion in the medical context. All clinical data, including genomic and proteomic, play a role in the diagnosis and in particular in the treatment planning and follow-up. This is true for all types of data analyses whether in classification, regression, retrieval, clustering, or other. The interaction between several types of information is not always well understood. Experienced clinicians automatically and even unconsciously add multiple sources of information into their decision process, but machine learning tools often concentrate on single information sources. This special issue presents five examples where several data sources are fused. The papers give several examples of fusion techniques and also the results obtained in quite different application scenarios.

Current developments of neuroimaging and genetics promote an integrative and compressive study of schizophrenia. However, it is still difficult to explore how gene mutations are related to brain abnormalities due to the high dimension but low sample size of these data. Conventional approaches reduce the dimension of dataset separately and then calculate the correlation, but ignore the effects of the response variables and the structure of data. To improve the identification of risk genes and abnormal brain regions on schizophrenia, in this paper, we propose a novel method called Independence and Structural sparsity Canonical Correlation Analysis (ISCCA). ISCCA combines independent component analysis (ICA) and Canonical Correlation Analysis (CCA) to reduce the collinear effects, which also incorporate graph structure of the data into the model to improve the accuracy of feature selection. The results from simulation studies demonstrate its higher accuracy in discovering correlations compared with other competing methods. Moreover, applying ISCCA to a real imaging genetics dataset collected by Mind Clinical Imaging Consortium (MCIC), a set of distinct gene-ROI interactions are identified, which are verified to be both statistically and biologically significant.

With the rapid development of high-throughput technologies, a growing amount of multi-omics data are collected, giving rise to a great demand for combining such data for biomedical discovery. Due to the cost and time to label the data manually, the number of labelled samples is limited. This motivated the need for semi-supervised learning algorithms. In this work, we applied a graph-based semi-supervised learning (GSSL) to classify a severe chronic mental disorder, schizophrenia (SZ). An advantage of GSSL is that it can simultaneously analyse more than two types of data, while many existing models focus on pairwise data analysis. In particular, we applied GSSL to the analysis of single nucleotide polymorphism (SNP), functional magnetic resonance imaging (fMRI) and DNA methylation data, which accounts for genetics, brain imaging (endophenotypes), and environmental factors (epigenomics) respectively. While parameter selection has been an open challenge for most models, another key contribution of this work is that we explored the parameter space to interpret their meaning and established practical guidelines. Based on the practical significance of each hyper-parameter, a relatively small range of candidate values can be determined in a data-driven way to both optimize and speed up the parameter tuning process. We validated the model through both synthetic data and a real SZ dataset of 184 subjects from the Mental Illness and Neuroscience Discovery (MIND) Clinical Imaging Consortium. In comparison to several existing approaches, our algorithm achieved better performance in terms of classification accuracy. We also confirmed the significance of several brain regions associated with SZ.

Human brain development is a complex and dynamic process caused by several factors such as genetics, sex hormones, and environmental changes. A number of recent studies on brain development have examined functional connectivity (FC) defined by the temporal correlation between time series of different brain regions. We propose to add the directional flow of information during brain maturation. To do so, we extract effective connectivity (EC) through Granger causality (GC) for two different groups of subjects, i.e., children and young adults. The motivation is that the inclusion of causal interaction may further discriminate brain connections between two age groups and help to discover new connections between brain regions. The contributions of this study are threefold. First, there has been a lack of attention to EC-based feature extraction in the context of brain development. To this end, we propose a new kernel-based GC (KGC) method to learn nonlinearity of complex brain network, where a reduced Sine hyperbolic polynomial (RSP) neural network was used as our proposed learner. Second, we used causality values as the weight for the directional connectivity between brain regions. Our findings indicated that the strength of connections was significantly higher in young adults relative to children. In addition, our new EC-based feature outperformed FC-based analysis from Philadelphia neurocohort (PNC) study with better discrimination of different age groups. Moreover, the fusion of these two sets of features (FC + EC) improved brain age prediction accuracy by more than 4%, indicating that they should be used together for brain development studies.

Canonical polyadic decomposition (CPD) of multi-subject complex-valued fMRI data can be used to provide spatially and temporally shared components among groups with both magnitude and phase information. However, the CPD model is not well formulated due to the large subject variability in the spatial and temporal modalities, as well as the high noise level in complexvalued fMRI data. Considering that the shift-invariant CPD can model temporal variability across subjects, we propose to further impose a phase sparsity constraint on the shared spatial maps to denoise the complex-valued components and to model the inter-subject spatial variability as well. More precisely, subject-specific time delays are first estimated for the complex-valued shared time courses in the framework of real-valued shift-invariant CPD. Source phase sparsity is then imposed on the complex-valued shared spatial maps. A smoothed ℓ0norm is specifically used to reduce voxels with large phase values after phase de-ambiguity based on the small phase characteristic of BOLD-related voxels. The results from both the simulated and experimental fMRI data demonstrate improvements of the proposed method over three complex-valued algorithms, namely, tensor-based spatial ICA, shift-invariant CPD and CPD without spatiotemporal constraints. When comparing with a real-valued algorithm combining shiftinvariant CPD and ICA, the proposed method detects 178.7% more contiguous task-related activations.

Recently, a hypergraph constructed from functional magnetic resonance imaging (fMRI) was utilized to explore brain functional connectivity networks (FCNs) for the classification of neurodegenerative diseases. Each edge of a hypergraph (called hyperedge) can connect any number of brain regions-of-interest (ROIs) instead of only two ROIs, and thus characterizes high-order relations among multiple ROIs that cannot be uncovered by a simple graph in the traditional graph based FCN construction methods. Unlike the existing hypergraph based methods where all hyperedges are assumed to have equal weights and only certain topological features are extracted from the hypergraphs, we propose a hypergraph learning based method for FCN construction in this paper. Specifically, we first generate hyperedges from fMRI time series based on sparse representation, then employ hypergraph learning to adaptively learn hyperedge weights, and finally define a hypergraph similarity matrix to represent the FCN. In our proposed method, weighting hyperedges results in better discriminative FCNs across subjects, and the defined hypergraph similarity matrix can better reveal the overall structure of brain network than using those hypergraph topological features. Moreover, we propose a multi-hypergraph learning based method by integrating multi-paradigm fMRI data, where the hyperedge weights associated with each fMRI paradigm are jointly learned and then a unified hypergraph similarity matrix is computed to represent the FCN. We validate the effectiveness of the proposed method on the Philadelphia Neurodevelopmental Cohort dataset for the classification of individuals' learning ability from three paradigms of fMRI data. Experimental results demonstrate that our proposed approach outperforms the traditional graph based methods (i.e., Pearson's correlation and partial correlation with the graphical Lasso) and the existing unweighted hypergraph based methods, which sheds light on how to optimize estimation of FCNs for cognitive and behavioral study.

Adolescence is a transitional period between the childhood and adulthood with physical changes, as well as increasing emotional development. Studies have shown that the emotional sensitivity is related to a second period of rapid brain growth. However, there is little focus on the trend of brain development during this period. In this paper, we aim to track functional brain connectivity development from late childhood to young adulthood. Mathematically, this problem can be modeled via the estimation of multiple Gaussian graphical models (GGMs). However, most existing methods either require the graph sequence to be fairly long or are only applicable to small graphs. In this paper, we adapted a Bayesian approach incorporating joint estimation of multiple GGMs to overcome the short sequence difficulty, which is also computationally efficient. The data used are the functional magnetic resonance imaging (fMRI) images obtained from the publicly available Philadelphia Neurodevelopmental Cohort (PNC). They include 855 individuals aged 8-22 years who were divided into five different adolescent stages. We summarized the networks with global measurements and applied a hypothesis test across age groups to detect the developmental patterns. Three patterns were detected and defined as consistent development, late puberty, and temporal change. We also discovered several anatomical areas, such as the middle frontal gyrus, putamen gyrus, right lingual gyrus, and right cerebellum crus 2 that are highly involved in the brain functional development. The functional networks, including the salience, subcortical, and auditory networks are significantly developing during the adolescent period.

Estimating dynamic functional network connectivity (dFNC) of the brain from functional magnetic resonance imaging (fMRI) data can reveal both spatial and temporal organization and can be applied to track the developmental trajectory of brain maturity as well as to study mental illness. Resting state fMRI (rs-fMRI) is regarded as a promising task since it reflects the spontaneous brain activity without an external stimulus. The sliding window method has been successfully used to extract dFNC but typically assumes a fixed window size. The hidden Markov model (HMM) based method is an alternative approach for estimating time-varying connectivity. In this paper, we propose a sparse HMM based on Gaussian HMM and Gaussian graphical model (GGM). In this model, the time-varying neural processes are represented as discrete brain states which are described with functional connectivity networks. By enforcing the sparsity on the precision matrix, we can get interpretable connectivity between different functional regions. The optimization of our model can be realized with the expectation maximization (EM) and graphical least absolute shrinkage and selection operator (glasso) algorithms. The proposed model is validated on both simulated blood oxygenation-level dependent (BOLD) time series and rs-fMRI data. Results indicate that the proposed model can capture both stationary and abrupt brain activity fluctuations. We also compare dFNC patterns between children and young adults from the Philadelphia Neurodevelopmental Cohort (PNC) study. Both spatial and temporal behavior of the dFNC are analyzed and compared. The results provide insight into the developmental trajectory across childhood and motivate further research on brain connectivity.

The birth of wireless communication systems nearly a century ago has transformed and redefined the way humans communicate and interact. This transformation has evolved over many years and has brought along not only seamless connectivity for human interactions but also communication between machines and devices. While these communication systems are manmade artifacts, the research community has more recently turned its attention to other communication strategies that have spontaneously evolved in nature.

Imaging genomics focuses on characterizing genomic influence on the variation of neurobiological traits, holding promise for illuminating the pathogenesis, reforming the diagnostic system, and precision medicine of mental disorders. This paper aims to provide an overall picture of the current status of neuroimaging-genomic analyses in mental disorders, and how we can increase their translational potential into clinical practice. The review is organized around three perspectives. (a) Towards reliability, generalizability and interpretability, where we summarize the multivariate models and discuss the considerations and trade-offs of using these methods and how reliable findings may be reached, to serve as ground for further delineation. (b) Towards improved diagnosis, where we outline the advantages and challenges of constructing a dimensional transdiagnostic model and how imaging genomic analyses map into this framework to aid in deconstructing heterogeneity and achieving an optimal stratification of patients that better inform treatment planning. (c) Towards improved treatment. Here we highlight recent efforts and progress in elucidating the functional annotations that bridge between genomic risk and neurobiological abnormalities, in detecting genomic predisposition and prodromal neurodevelopmental changes, as well as in identifying imaging genomic biomarkers for predicting treatment response. Providing an overview of the challenges and promises, this review hopefully motivates imaging genomic studies with multivariate, dimensional and transdiagnostic designs for generalizable and interpretable findings that facilitate development of personalized treatment.

Schizophrenia (SZ) is a chronic and severe mental disorder that affects how a person thinks, feels, and behaves. It has been proposed that this disorder is related to disrupted brain connectivity, which has been verified by many studies. With the development of functional magnetic resonance imaging (fMRI), further exploration of brain connectivity was made possible. Region-based networks are commonly used for mapping brain connectivity. However, they fail to illustrate the connectivity within regions of interest (ROIs) and lose precise location information. Voxel-based networks provide higher precision, but are difficult to construct and interpret due to the high dimensionality of the data. In this paper, we adopt a novel high-dimensional Gaussian graphical model - ψ-learning method, which can help ease computational burden and provide more accurate inference for the underlying networks. This method has been proven to be an equivalent measure of the partial correlation coefficient and, thus, is flexible for network comparison through statistical tests. The fMRI data we used were collected by the mind clinical imaging consortium using an auditory task in which there are 92 SZ patients and 116 healthy controls. We compared the networks at three different scales by using global measurements, community structure, and edge-wise comparisons within the networks. Our results reveal, at the highest voxel resolution, sets of distinct aberrant patterns for the SZ patients, and more precise local structures are provided within ROIs for further investigation.

Dynamic functional connectivity analysis is an effective way to capture the networks that are functionally associated and continuously changing over the scanning period. However, these methods mostly analyze the dynamic associations across the activation patterns of the spatial networks while assuming that the spatial networks are stationary. Hence, a model that allows for the variability in both domains and reduces the assumptions imposed on the data provides an effective way for extracting spatiotemporal networks. Independent vector analysis (IVA) is a joint blind source separation technique that allows for estimation of spatial and temporal features while successfully preserving variability. However, its performance is affected for higher number of datasets. Hence, we develop an effective two-stage method to extract time-varying spatial and temporal features using IVA, mitigating the problems with higher number of datasets while preserving the variability across subjects and time. The first stage is used to extract reference signals using group-independent component analysis (GICA) that are used in a parameter-tuned constrained IVA framework to estimate time-varying representations of these signals by preserving the variability through tuning the constraint parameter. This approach effectively captures variability across time from a large-scale resting-state fMRI data acquired from healthy controls and patients with schizophrenia and identifies more functionally relevant connections that are significantly different among healthy controls and patients with schizophrenia, compared with the widely used GICA method alone.

Methods based on independent component analysis (ICA) and canonical correlation analysis (CCA) as well as their various extensions have become popular for the fusion of multimodal data as they minimize assumptions about the relationships among multiple datasets. Two important extensions that are widely used, joint ICA (jICA) and parallel ICA (pICA), make a number of simplifying assumptions that might limit their usefulness such as identical mixing matrices for jICA, and the requirement for the same number of components for jICA and pICA. In this paper, we propose a new, flexible hybrid method for fusion based on ICA and CCA, called consecutive independence and correlation transform (C-ICT), which relaxes the main limitations of jICA and pICA. We demonstrate performance advantages of C-ICT both through simulations and application to real medical data collected from schizophrenia patients and healthy controls performing an auditory oddball task (AOD).

Recently, much attention has been devoted to examining time-varying changes in functional connectivity to understand the network structure in the human brain. Most studies, however, analyze the time-varying functional connectivity but ignore the time-varying spatial information. In this paper, we propose a method based on independent vector analysis (IVA) to study dynamic functional network connectivity (dFNC) as well as dynamic spatial functional network connectivity (dsFNC) in fMRI data. Though IVA allows one to effectively capture both, its performance degrades with the increase in the number of datasets. Hence, we propose an effective scheme to bypass this limitation followed by graph theoretical analysis to study both inter-network dynamics and intra-network stationarity. We observe higher dFNC fluctuations for patients with schizophrenia in the default-mode (DM)-salience network and cerebellum with associated connections. dsFNC analysis indicates higher inter-network fluctuation in patients while DM, anterior DM and frontal networks demonstrate significant intra-network fluctuation in controls.

Dynamic functional connectivity has become a prominent approach for tracking the changes of macroscale statistical dependencies between regions in the brain. Effective parametrization of these statistical dependencies, referred to as brain states, is however still an open problem. We investigate different emission models in the hidden Markov model framework, each representing certain assumptions about dynamic changes in the brain. We evaluate each model by how well they can discriminate between schizophrenic patients and healthy controls based on a group independent component analysis of resting-state functional magnetic resonance imaging data. We find that simple emission models without full covariance matrices can achieve similar classification results as the models with more parameters. This raises questions about the predictability of dynamic functional connectivity in comparison to simpler dynamic features when used as biomarkers. However, we must stress that there is a distinction between characterization and classification, which has to be investigated further.

Trends, they are not only for the fashion industry after all. Within the engineering and computer science research communities as well, we periodically observe the phenomenon, see how certain methods suddenly start receiving particular attention, and sometimes, even though they emerge as an attractive solution for a given set of problems, they tend to become a hammer looking for new nails. At fi rst, using a new method on old problems is the natural and reasonable way to proceed. There have been remarkable successes achieved through the adoption of a tool from another fi eld or a new way of looking at old problems that brings new insights and solutions. There have been a number of such trends throughout the years in every field. In signal processing, a few notable ones include maximum entropy, wavelets, kernel methods, and the multiple up and down cycles of neural nets. A current tool that is on the rise is sparsity, more specifically solutions that promote sparsity, including coding, compressive sensing, sparse learning/estimation, sparse factorizations, and of course deep nets, which, without careful use of sparsity, would be useless. We will use sparsity as the major example in this Point of View article because it is current and illustrates the points we are making very well. While we agree that sparsity is very useful and has led to some excellent results in the past decade or so, it also allows us to address the dangers and pitfalls of blindly following trends in research. These problems are reflected in our publications and help define the overall research climate. In the following, we provide an overview on use of sparsity and then discuss a couple of specific problems.

Human brain connectivity is complex. Graph theory based analysis has become a powerful and popular approach for analyzing brain imaging data, largely because of its potential to quantitatively illuminate the networks, the static architecture in structure and function, the organization of dynamic behavior over time, and disease related brain changes. The first step in creating brain graphs is to define the nodes and edges connecting them. We review a number of approaches for defining brain nodes including fixed versus data-driven nodes. Expanding the narrow view of most studies which focus on static and/or single modality brain connectivity, we also survey advanced approaches and their performances in building dynamic and multi-modal brain graphs. We show results from both simulated and real data from healthy controls and patients with mental illnesse. We outline the advantages and challenges of these various techniques. By summarizing and inspecting recent studies which analyzed brain imaging data based on graph theory, this article provides a guide for developing new powerful tools to explore complex brain networks.

It's increasingly important but difficult to determine potential biomarkers of schizophrenia (SCZ) disease, owing to the complex pathophysiology of this disease. In this study, a network-fusion based framework was proposed to identify genetic biomarkers of the SCZ disease. A three-step feature selection was applied to single nucleotide polymorphisms (SNPs), DNA methylation, and functional magnetic resonance imaging (fMRI) data to select important features, which were then used to construct two gene networks in different states for the SNPs and DNA methylation data, respectively. Two health networks (one is for SNP data and the other is for DNA methylation data) were combined into one health network from which health minimum spanning trees (MSTs) were extracted. Two disease networks also followed the same procedures. Those genes with significant changes were determined as SCZ biomarkers by comparing MSTs in two different states and they were finally validated from five aspects. The effectiveness of the proposed discovery framework was also demonstrated by comparing with other network-based discovery methods. In summary, our approach provides a general framework for discovering gene biomarkers of the complex diseases by integrating imaging genomic data, which can be applied to the diagnosis of the complex diseases in the future.

Recent advances in imaging genetics produce large amounts of data including functional MRI images, single nucleotide polymorphisms (SNPs), and cognitive assessments. Understanding the complex interactions among these heterogeneous and complementary data has the potential to help with diagnosis and prevention of mental disorders. However, limited efforts have been made due to the high dimensionality, group structure, and mixed type of these data. In this paper we present a novel method to detect conditional associations between imaging genetics data. We use projected distance correlation to build a conditional dependency graph among high-dimensional mixed data, then use multiple testing to detect significant group level associations (e.g., ROI-gene). In addition, we introduce a scalable algorithm based on orthogonal greedy algorithm, yielding the greedy projected distance correlation (G-PDC). This can reduce the computational cost, which is critical for analyzing large-volume of imaging genomics data. The results from our simulations demonstrate a higher degree of accuracy with GPDC than distance correlation, Pearson's correlation and partial correlation, especially when the correlation is nonlinear. Finally, we apply our method to the Philadelphia Neurodevelopmental data cohort with 866 samples including fMRI images and SNP profiles. The results uncover several statistically significant and biologically interesting interactions, which are further validated with many existing studies. The Matlab code is available at https://sites.google.com/site/jianfang86/gPDC.

Functional connectivity (FC) estimated from functional magnetic resonance imaging (fMRI) time series, especially during resting state periods, provides a powerful tool to assess human brain functional architecture in health, disease, and developmental states. Recently, the focus of connectivity analysis has shifted toward the subnetworks of the brain, which reveals co-activating patterns over time. Most prior works produced a dense set of high-dimensional vectors, which are hard to interpret. In addition, their estimations to a large extent were based on an implicit assumption of spatial and temporal stationarity throughout the fMRI scanning session. In this paper, we propose an approach called dynamic sparse connectivity patterns (dSCPs), which takes advantage of both matrix factorization and time-varying fMRI time series to improve the estimation power of FC. The feasibility of analyzing dynamic FC with our model is first validated through simulated experiments. Then, we use our framework to measure the difference between young adults and children with real fMRI data set from the Philadelphia Neurodevelopmental Cohort (PNC). The results from the PNC data set showed significant FC differences between young adults and children in four different states. For instance, young adults had reduced connectivity between the default mode network and other subnetworks, as well as hyperconnectivity within the visual system in states 1 and 3, and hypoconnectivity in state 2. Meanwhile, they exhibited temporal correlation patterns that changed over time within functional subnetworks. In addition, the dSCPs model indicated that older people tend to spend more time within a relatively connected FC pattern. Overall, the proposed method provides a valid means to assess dynamic FC, which could facilitate the study of brain networks.

Reducing the number of false discoveries is presently one of the most pressing issues in the life sciences. It is of especially great importance for many applications in neuroimaging and genomics, where data sets are typically high-dimensional, which means that the number of explanatory variables exceeds the sample size. The false discovery rate (FDR) is a criterion that can be employed to address that issue. Thus it has gained great popularity as a tool for testing multiple hypotheses. Canonical correlation analysis (CCA) is a statistical technique that is used to make sense of the cross-correlation of two sets of measurements collected on the same set of samples (e.g., brain imaging and genomic data for the same mental illness patients), and sparse CCA extends the classical method to high-dimensional settings. Here, we propose a way of applying the FDR concept to sparse CCA, and a method to control the FDR. The proposed FDR correction directly influences the sparsity of the solution, adapting it to the unknown true sparsity level. Theoretical derivation as well as simulation studies show that our procedure indeed keeps the FDR of the canonical vectors below a user-specified target level. We apply the proposed method to an imaging genomics data set from the Philadelphia Neurodevelopmental Cohort. Our results link the brain connectivity profiles derived from brain activity during an emotion identification task, as measured by functional magnetic resonance imaging, to the corresponding subjects' genomic data.

By exploiting cross-information among multiple imaging data, multimodal fusion has often been used to better understand brain diseases. However, most current fusion approaches are blind, without adopting any prior information. There is increasing interest to uncover the neurocognitive mapping of specific clinical measurements on enriched brain imaging data; hence, a supervised, goal-directed model that employs prior information as a reference to guide multimodal data fusion is much needed and becomes a natural option. Here, we proposed a fusion with reference model called "multi-site canonical correlation analysis with reference + joint-independent component analysis" (MCCAR+jICA), which can precisely identify co-varying multimodal imaging patterns closely related to the reference, such as cognitive scores. In a three-way fusion simulation, the proposed method was compared with its alternatives on multiple facets; MCCAR+jICA outperforms others with higher estimation precision and high accuracy on identifying a target component with the right correspondence. In human imaging data, working memory performance was utilized as a reference to investigate the co-varying working memory-associated brain patterns among three modalities and how they are impaired in schizophrenia. Two independent cohorts (294 and 83 subjects respectively) were used. Similar brain maps were identified between the two cohorts along with substantial overlaps in the central executive network in fMRI, salience network in sMRI, and major white matter tracts in dMRI. These regions have been linked with working memory deficits in schizophrenia in multiple reports and MCCAR+jICA further verified them in a repeatable, joint manner, demonstrating the ability of the proposed method to identify potential neuromarkers for mental disorders.

In this paper, we consider the problem of estimating multiple sparse, co-activated brain regions from functional magnetic resonance imaging (fMRI) observations belonging to different classes. More precisely, we propose a method to analyze similarities and differences in functional connectivity between children and young adults. Often, analysis is conducted on each class separately, and differences across classes are identified with an additional postprocessing step using adequate statistical tools. Here, we propose to rely on a generalized fused Lasso penalty, which allows us to make use of the entire data set in order to estimate connectivity patterns that are either shared across classes, or specific to a given group. By using the entire population during the estimation, we hope to increase the power of our analysis. The proposed model falls in the category of population-wise matrix decomposition, and a simple and efficient alternating direction method of multipliers algorithm is introduced to solve the associated optimization problem. After validating our approach on simulated data, experiments are performed on resting-state fMRI imaging from the Philadelphia neurodevelopmental cohort data set, comprised of normally developing children from ages 8 to 21. Developmental differences were observed in various brain regions, as a total of three class-specific resting-state components were identified. Statistical analysis of the estimated subject-specific features, as well as classification results (based on age groups, up to 81% accuracy, samples) related to these components demonstrate that the proposed method is able to properly extract meaningful shared and class-specific sub-networks.

Among the challenges arising in brain imaging genetic studies, estimating the potential links between neurological and genetic variability within a population is key. In this paper, we propose a multivariate, multimodal formulation for variable selection that leverages co-expression patterns across various data modalities. Our approach is based on an intuitive combination of two widely used statistical models: sparse regression and canonical correlation analysis (CCA). While the former seeks multivariate linear relationships between a given phenotype and associated observations, the latter searches to extract co-expression patterns between sets of variables belonging to different modalities. In the following, we propose to rely on a "CCA-type" formulation in order to regularize the classical multimodal sparse regression problem (essentially incorporating both CCA and regression models within a unified formulation). The underlying motivation is to extract discriminative variables that are also co-expressed across modalities. We first show that the simplest formulation of such model can be expressed as a special case of collaborative learning methods. After discussing its limitation, we propose an extended, more flexible formulation, and introduce a simple and efficient alternating minimization algorithm to solve the associated optimization problem. We explore the parameter space and provide some guidelines regarding parameter selection. Both the original and extended versions are then compared on a simple toy data set and a more advanced simulated imaging genomics data set in order to illustrate the benefits of the latter. Finally, we validate the proposed formulation using single nucleotide polymorphisms data and functional magnetic resonance imaging data from a population of adolescents ( subjects, age 16.9 ± 1.9 years from the Philadelphia Neurodevelopmental Cohort) for the study of learning ability. Furthermore, we carry out a significance analysis of the resulting features that allow us to carefully extract brain regions and genes linked to learning and cognitive ability.

The existence of complementary information across multiple sensors has driven the proliferation of multivariate datasets. Exploitation of this common information, while minimizing the assumptions imposed on the data has led to the popularity of data-driven methods. Independent vector analysis (IVA), in particular, provides a flexible and effective approach for the fusion of multivariate data. In many practical applications, important prior information about the data exists and incorporating this information into the IVA model is expected to yield improved separation performance. In this paper, we propose a general formulation for non-orthogonal constrained IVA (C-IVA) framework that can incorporate prior information about either the sources or the mixing coefficients into the IVA cost function. A powerful decoupling method is the major enabling factor in this task. We demonstrate the improved performance of C-IVA over the unconstrained IVA model using both simulated as well as real medical imaging data.

While the majority of exploratory approaches search for correlations among features of different modalities, indirect/nonlinear relations between structure and function have not yet been fully investigated. In this work, we employ a neural machine translation model [1] to relate two modalities: structural MRI (sMRI) spatial components and functional MRI (fMRI) brain states estimated using a dynamic connectivity model. We consider each of the modalities as different “languages” of the same brain and fit a translation model to estimate a model for how structure influences function. Results identify multiple aligned aspects of brain structure and functional brain states showing significantly more or less alignment in the patient group as well as interesting links to other variables such as cognitive scores and symptom assessments. Our novel approach provides a new perspective on combining brain structure and function by incorporating indirect/nonlinear effects and enabling the algorithm to learn the interplay between structural and the functional networks.

Functional magnetic resonance imaging (fMRI) has provided a window into the brain with wide adoption in research and even clinical settings. Data-driven methods such as those based on latent variable models and matrix/tensor factorizations are being increasingly used for fMRI data analysis. There is increasing availability of large-scale multi-subject repositories involving 1,000+ individuals. Studies with large numbers of data sets promise effective comparisons across different conditions, groups, and time points, further increasing the utility of fMRI in human brain research. In this context, there is a pressing need for innovative ideas to develop flexible analysis methods that can scale to handle large-volume fMRI data, process the data in a distributed and policy-compliant manner, and capture diverse global and local patterns leveraging the big pool of fMRI data. This paper is a survey of some of the recent research in this direction.

Magnitude-only resting-state fMRI data have been largely investigated via independent component analysis (ICA) for exacting spatial maps (SMs) and time courses. However, the native complex-valued fMRI data have rarely been studied. Motivated by the significant improvements achieved by ICA of complex-valued task fMRI data than magnitude-only task fMRI data, we present an efficient method for de-noising SM estimates which makes full use of complex-valued resting-state fMRI data. Our two main contributions include: (1) The first application of a post-ICA phase de-noising method, originally proposed for task fMRI data, to resting-state data, which recognizes voxels within a specific phase range as desired voxels. (2) A new phase range detection strategy for a specific SM component based on correlation with its reference. We continuously change the phase range within a larger range, and compute a set of correlation coefficients between each de-noised SM and its reference. The phase range with the maximal correlation determines the final selection. The detected results by the proposed approach confirm the correctness of the post-ICA phase de-noising method in the analysis of resting-state complex-valued fMRI data.

Exploitation of complementary information is the principal reason for collecting data from multiple neurological sensors. Since little is known about the latent processes underlying neural function, it is important to minimize the assumptions placed on the data when performing a joint analysis. This motivates the use of data-driven fusion methods, such as independent vector analysis (IVA), for the analysis of neurological data. For neural datasets, the complementary information exploited by fusion methods may be in the form of similar spatial activation across datasets, the spatial IVA (sIVA) model, or similar subject relations across datasets, the transposed IVA (tIVA) model. Despite the potential power of these two models, no study has investigated how the differences in the modeling assumptions of sIVA and tIVA inform the fusion of real neuro-imaging data. In this paper, we utilize a unique set of multitask functional magnetic resonance imaging data from 271 subjects to directly compare the sIVA and tIVA models and visualize their differences using a novel technique, global difference maps. Through this application, we note important similarities between the results from the two methods that increase our confidence in their overall performance, though differences in modeling assumptions result in certain differences in the decompositions.

Numerous studies have shown that brain functional connectivity patterns can be time-varying over periods of tens of seconds. It is important to capture inherent non-stationary connectivity states for a better understanding of the influence of disease on brain connectivity. K-means has been widely used to extract the connectivity states from dynamic functional connectivity. However, K-means is dependent on initialization and can be exponentially slow in converging due to extensive noise in dynamic functional connectivity. In this work, we propose to use an affinity propagation clustering method to estimate the connectivity states. By applying K-means and the new method separately, we analyzed dynamic functional connectivity of 82 healthy controls and 82 schizophrenia patients, and then explored group differences between schizophrenia patients and healthy controls in the identified connectivity states. Both methods revealed that group differences mainly lay in visual, sensorimotor and frontal cortices. However, the new approach found more meaningful group differences than K-means. Our finding supports that our method is promising in exploring biomarkers of mental disorders.

Schizophrenia (SZ) is a complex disease caused by a lot genetic variants, epigenetic and brain region abnormalities. In this study, we adopted a joint nonnegative matrix factorization method to integrate three datasets including single nucleotide polymorphism (SNP), brain activity measured by functional magnetic resonance imaging (fMRI) and DNA Methylation to identify multi-dimensional modules associated with SZ. They are then used to study the coordination between regulatory mechanisms at multiple levels. This method projects multiple types of data onto a common feature space, in which heterogeneous variables with large coefficients on the same projected bases form a multi-dimensional module. The genomic factors in such modules have significant correlations and likely functional associations with brain activities. We applied this method to the real data analysis and identified multi-dimensional modules including SNP, fMRI and DNA methylation sites. These selected biomarkers were finally used to identify genes and voxels, which were confirmed to be significantly associated with SZ.

Independent vector analysis (IVA) is an approach for joint blind source separation of several data sets that learns simultaneous unmixing transforms for each set. It assumes corresponding sources from different data sets to be statistically dependent. One of the main advantages is IVA's ability to retain subject-specific differences while simplifying comparison across subjects as the resulting components have the same order. The latter is an instrumental property for enabling collaboration between remote sites without sharing their data, which may be required because of ethical, privacy or efficiency concerns. This paper proposes a new decentralized algorithm for IVA that exploits the structure of the objective function. A centralized aggregator coordinates IVA algorithms at multiple sites using message passing, parallelizing the computation and limiting the amount of communication. Thus, the algorithm enables a plausibly private collaboration across multiple sites. Besides enabling analysis of decentralized data, our approach improves the running time of IVA when used locally.

Functional magnetic resonance imaging (fMRI) is a powerful tool to analyze brain development and neuronal activity. Identifying discriminative brain regions between various groups within a population has generated great interest in recent years. In this work, we consider the problem of estimating multiple sparse, co-activated brain regions from fMRI observations belonging to different classes. More precisely, we propose a method to analyze functional connectivity differences between children and young adults. Often, analysis is conducted on each class separately. Here, we propose to rely on a generalized fused Lasso penalty to extract both class-specific and shared co-expressed regions. In order to validate our method, experiments are performed on an fMRI dataset comprised of normally developing children from 8 to 21. The results demonstrate that the proposed method is able to properly extract meaningful sub-networks, which results in improved classification accuracy between the two classes.

Visualization of high dimensional large-scale datasets via an embedding into a 2D map is a powerful exploration tool for assessing latent structure in the data and detecting outliers. There are many methods developed for this task but most assume that all pairs of samples are available for common computation. Specifically, the distances between all pairs of points need to be directly computable. In contrast, we work with sensitive neuroimaging data, when local sites cannot share their samples and the distances cannot be easily computed across the sites. Yet, the desire is to let all the local data participate in collaborative computation without leaving their respective sites. In this scenario, a quality control tool that visualizes decentralized dataset in its entirety via global aggregation of local computations is especially important as it would allow screening of samples that cannot be evaluated otherwise. This paper introduces an algorithm to solve this problem: decentralized data stochastic neighbor embedding (dSNE). Based on the MNIST dataset we introduce metrics for measuring the embedding quality and use them to compare dSNE to its centralized counterpart. We also apply dSNE to a multi-site neuroimaging dataset with encouraging results.

Segmenting a structural magnetic resonance imaging (MRI) scan is an important pre-processing step for analytic procedures and subsequent inferences about longitudinal tissue changes. Manual segmentation defines the current gold standard in quality but is prohibitively expensive. Automatic approaches are computationally intensive, incredibly slow at scale, and error prone due to usually involving many potentially faulty intermediate steps. In order to streamline the segmentation, we introduce a deep learning model that is based on volumetric dilated convolutions, subsequently reducing both processing time and errors. Compared to its competitors, the model has a reduced set of parameters and thus is easier to train and much faster to execute. The contrast in performance between the dilated network and its competitors becomes obvious when both are tested on a large dataset of unprocessed human brain volumes. The dilated network consistently outperforms not only another state-of-the-art deep learning approach, the up convolutional network, but also the ground truth on which it was trained. Not only can the incredible speed of our model make large scale analyses much easier but we also believe it has great potential in a clinical setting where, with little to no substantial delay, a patient and provider can go over test results.

Researchers often wish to study data stored in separate locations, such as when several research entities wish to make inferences from their combined data. The most common solution is to centralize the data in one location. However, certain types of data can be difficult to transfer between entities due to legal or practical reasons. This makes centralizing these types of data problematic. A possible solution is the use of methods that learn from data without moving them to a central location: decentralized algorithms. Only a few algorithms emphasizing that property are known to us, and even fewer are used in the biomedical domain. In this paper, we propose a decentralized neural network that allows data analysis without transferring the data from the sites that host them. Instead, this method only transfers the gradients (or their parts) calculated via back-propagation. Our approach allows us to learn a classifier even when class examples are located at different sites, enabling privacy-aware collaboration across groups with specific research interests. We validate the method in several experiments to test stability, compare performance to a network trained on the centralized data, and investigate the ability to reduce size of data transfer. Our experiments on simulated, benchmark, and neuroimaging addiction data provide strong evidence that the proposed model works as effectively as a pooled centralized model.

Neuroimaging modalities such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide information about neurological functions in complementary spatiotemporal resolutions; therefore, fusion of these modalities is expected to provide better understanding of brain activity. In this paper, we jointly analyze fMRI and EEG data collected during an auditory oddball task with the goal of capturing brain activity patterns that differ between patients with schizophrenia and healthy controls. Rather than selecting a single electrode or matricizing the third-order tensor that can be naturally used to represent multi-channel EEG signals, we preserve the multi-way structure of EEG data and use a coupled matrix and tensor factorization (CMTF) model to jointly analyze fMRI and EEG signals. Our analysis reveals that (i) joint analysis of EEG and fMRI using a CMTF model can capture meaningful temporal and spatial signatures of patterns that behave differently in patients and controls, and (ii) these differences and the interpretability of the associated components increase by including multiple electrodes from frontal, motor and parietal areas, but not necessarily by including all electrodes in the analysis.

Human decision making is a multidimensional construct, driven by a complex interplay between external factors, internal biases, and computational capacity constraints. Here, we propose a layered approach to experimental design in which multiple tasks-from simple to complex-with additional layers of complexity introduced at each stage are incorporated for investigating decision making. This is demonstrated using tasks involving intertemporal choice between immediate and future prospects. Previous functional magnetic resonance imaging (fMRI) and electroencephalographic (EEG) studies have separately investigated the spatial and temporal neural substrates, respectively, of specific factors underlying decision making. In contrast, we performed simultaneous acquisition of EEG/fMRI data and fusion of both modalities using joint independent component analysis such that: 1) the native temporal/spatial resolutions of either modality is not compromised and 2) fast temporal dynamics of decision making as well as involved deeper striatal structures can be characterized. We show that spatiotemporal neural substrates underlying our proposed complex intertemporal task simultaneously incorporating rewards, costs, and uncertainty of future outcomes can be predicted (using a linear model) from neural substrates of each of these factors, which were separately obtained by simpler tasks. This was not the case for spatial and temporal features obtained separately from fMRI and EEG, respectively. However, certain prefrontal activations in the complex task could not be predicted from activations in simpler tasks, indicating that the assumption of pure insertion has limited validity. Overall, our approach provides a realistic and novel framework for investigating the neural substrates of decision making with high spatiotemporal resolution.

The extraction of information from multiple sets of data is a problem inherent to many disciplines. This is possible by either analyzing the data sets jointly as in data fusion or separately and then combining as in data integration. However, selecting the optimal method to combine and analyze multiset data is an ever-present challenge. The primary reason for this is the difficulty in determining the optimal contribution of each data set to an analysis as well as the amount of potentially exploitable complementary information among data sets. In this paper, we propose a novel classification rate-based technique to unambiguously quantify the contribution of each data set to a fusion result as well as facilitate direct comparisons of fusion methods on real data and apply a new method, independent vector analysis (IVA), to multiset fusion. This classification rate-based technique is used on functional magnetic resonance imaging data collected from 121 patients with schizophrenia and 150 healthy controls during the performance of three tasks. Through this application, we find that though optimal performance is achieved by exploiting all tasks, each task does not contribute equally to the result and this framework enables effective quantification of the value added by each task. Our results also demonstrate that data fusion methods are more powerful than data integration methods, with the former achieving a classification rate of 73.5 % and the latter achieving one of 70.9 %, a difference which we show is significant when all three tasks are analyzed together. Finally, we show that IVA, due to its flexibility, has equivalent or superior performance compared with the popular data fusion method, joint independent component analysis.

In order to increase the accuracy for the diagnosis of schizophrenia (SCZ) disease, it is essential to integratively employ complementary information from multiple types of data. It is well known that a network is a graph based method for analyzing relationships between patients, with its nodes and edges representing patients and their relationships respectively. In this study, we developed a network-based prediction approach by taking advantage of fused network from multiple data types rather than individual networks. Specifically, we constructed a fused network using three types of data including genetic, epigenetic and neuroimaging data from the study of schizophrenia. The majority neighborhood of a node in the network was exploited for discriminating SCZ from healthy controls. In comparison with other 9 graph-based label prediction methods, our prediction method shows the best performance according to several metrics. The prediction power of our proposed method was also tested with different parameters and optimal parameters were determined. We show that the label prediction method based on network fusion from multiple data types shows promises for more accurate diagnosis of schizophrenia, which can also be extended to other disease models.

Schizophrenia (SCZ) disease ranks among the top 10 causes of disability in developed countries worldwide. Its onset is the combination result of genetic, biological and environmental factors. It is increasingly important but difficult to determine which genes are potential biomarkers for SCZ, owing to the complex nature of the pathophysiology of this disease. In our study, we integrated genomic, epigenomic and neuroimaging data to identify genetic biomarkers for schizophrenia. Important cross-correlated features were selected using multiple sparse canonical correlation analysis (smCCA) among single nucleotide polymorphism (SNP), DNA methylation and functional magnetic resonance imaging (fMRI) data. The features were then used to construct two state (health and case) gene-gene interaction networks for SNP or DNA methylation data. A network-based framework was proposed by comparing two different minimum spanning trees (MSTs), which were extracted from two fused state gene networks, respectively. We selected top 20 genes with significant changes of network features for schizophrenia. These genes were finally validated by disease association enrichment analysis, Gene Ontology (GO) enrichment analysis, pathway enrichment analysis and related literature reports. We also demonstrated the effectiveness of our framework through the comparison with other network-based discovery methods. Therefore, our proposed network-based approach can effectively discover biomarkers and resulting genes, promising better diagnosis and treatment of schizophrenia disease.

Kernel and Multiple Kernel Canonical Correlation Analysis (CCA) are employed to classify schizophrenic and healthy patients based on their SNPs, DNA Methylation and fMRI data. Kernel and Multiple Kernel CCA are popular methods for finding nonlinear correlations between high-dimensional datasets. Data was gathered from 183 patients, 79 with schizophrenia and 104 healthy controls. Kernel and Multiple Kernel CCA represent new avenues for studying schizophrenia, because, to our knowledge, these methods have not been used on these data before. Classification is performed via k nearest neighbors on the kernel matrix outputs of the Kernel and Multiple Kernel CCA algorithm. Accuracies of the Kernel and Multiple Kernel CCA classification are compared to that of the regularized linear CCA algorithm classification, and are found to be significantly more accurate. Both algorithms demonstrate maximal accuracies when the combination of DNA methylation and fMRI data are used, and experience lower accuracies when the SNP data are incorporated.

Independent vector analysis (IVA) has exhibited great potential for the group analysis of magnitude-only fMRI data, but has rarely been applied to native complex-valued fMRI data. We propose an adaptive fixed-point IVA algorithm by taking into account the extremely noisy nature, large variability of the source component vector (SCV) distribution, and non-circularity of the complex-valued fMRI data. The multivariate generalized Gaussian distribution (MGGD) is exploited to match the SCV distribution based on nonlinearity, the shape parameter of MGGD is estimated using maximum likelihood estimation, and the nonlinearity is updated in the dominant SCV subspace to achieve denoising goal. In addition, the pseudo-covariance matrix is incorporated into the algorithm to represent the non-circularity. Experimental results from simulated and actual fMRI data demonstrate significant improvements of our algorithm over a complex-valued IVA-G algorithm and several circular and noncircular fixed-point IVA variants.

Spectral analysis of brain activation in different regions, either in the form of network time-courses or regions of interest (ROI) time-series, has been a topic of interest in recent studies. Such studies hypothesize that observed brain fluctuations are due to different underlying sources of neurophysiological activation. Among these studies, brain fluctuations during the resting-state, as an unconstrained condition, have been a subject of interest. Some clinical studies have employed spectral analysis to locate differences between diagnostic groups such as schizophrenia and bipolar disorder. Other studies have argued that resting-state brain fluctuations are in fact dynamic, and that activation and connectivity of brain regions develops and evolves spontaneously. In this study, we combine both approaches and focus on capturing dynamics of the spectral properties of network time-courses estimated from independent components analysis (ICA) and categorizing spontaneous frequency profiles of network time-courses into three major profiles, which we call "frequency modes". We show that brain networks have distinct time-varying frequency domain characteristics, differing from one another in their occupancy rates of the frequency modes. Additionally, we identify some networks in which the occurrence rates of the different modes are significantly different based on the gender of the subjects.

Variational methods that rely on a recognition network to approximate the posterior of directed graphical models offer better inference and learning than previous methods. Recent advances that exploit the capacity and flexibility in this approach have expanded what kinds of models can be trained. However, as a proposal for the posterior, the capacity of the recognition network is limited, which can constrain the representational power of the generative model and increase the variance of Monte Carlo estimates. To address these issues, we introduce an iterative refinement procedure for improving the approximate posterior of the recognition network and show that training with the refined posterior is competitive with state-of-the-art methods. The advantages of refinement are further evident in an increased effective sample size, which implies a lower variance of gradient estimates.

MOTIVATION: Imaging genetics combines brain imaging and genetic information to identify the relationships between genetic variants and brain activities. When the data samples belong to different classes (e.g. disease status), the relationships may exhibit class-specific patterns that can be used to facilitate the understanding of a disease. Conventional approaches often perform separate analysis on each class and report the differences, but ignore important shared patterns. RESULTS: In this paper, we develop a multivariate method to analyze the differential dependency across multiple classes. We propose a joint sparse canonical correlation analysis method, which uses a generalized fused lasso penalty to jointly estimate multiple pairs of canonical vectors with both shared and class-specific patterns. Using a data fusion approach, the method is able to detect differentially correlated modules effectively and efficiently. The results from simulation studies demonstrate its higher accuracy in discovering both common and differential canonical correlations compared to conventional sparse CCA. Using a schizophrenia dataset with 92 cases and 116 controls including a single nucleotide polymorphism (SNP) array and functional magnetic resonance imaging data, the proposed method reveals a set of distinct SNP-voxel interaction modules for the schizophrenia patients, which are verified to be both statistically and biologically significant. AVAILABILITY AND IMPLEMENTATION: The Matlab code is available at https://sites.google.com/site/jianfang86/JSCCA CONTACT: [email protected] information: Supplementary data are available at Bioinformatics online.

Patterns of resting state fMRI functional network connectivity in schizophrenia patients have been shown to differ markedly from those of healthy controls. While some studies have explored connectivity within fixed frequency bands, the question of network phase synchrony across disparate frequency bands, or cross-frequency connectivity , has remained surprisingly underexplored. Computational modeling at the neuronal scale however has long acknowledged the existence of coupled fast and slow subsystems. Here, we present preliminary evidence that cross-frequency coupling exists at the network level, that it patterns in meaningful ways over functional domains, and that this patterning differs between the healthy population and individuals with diagnosed schizophrenia.

Multiple types of signals or images are often collected from the same participants in biomedical research. Multimodal analyses have been shown to better capture the joint information. We propose a new method named parallel group independent component analysis (para-GICA) to address a special need for parallel processing of multimodal brain images or signals where it is desirable to partition into groups, for example to stratify by age. Para-GICA is designed to identify associated components between two modalities based on their loading variations in participants, while allowing components to show group specificity. Simulation using synthetic MRI and genetic data demonstrates that para-GICA is able to recover group specific brain networks and the connection between brain networks and genetic factors. A real data application on brain gray matter concentration and whiter matter fractional anisotropy images extracts associated gray matter and white matter components, and ageing induced spatial differences of the components.

Schizophrenia is currently diagnosed by physicians through evaluation of their clinical assessment and a patient's self-reported experience over the longitudinal course of the illness. There is great interest in identifying biologically based markers at illness onset, rather than relying on the evolution of symptoms across time. Functional connectivity shows promise in providing individual subject predictive power. However, the majority of previous studies considered only the analysis of functional connectivity during resting-state or performance of a single task. Changes in connectivity between rest and multiple tasks have not been used in the discrimination of schizophrenia patients from healthy controls. In this work, we propose a framework for classification of schizophrenia patients and healthy control subjects based on functional network component pairs which show consistency between patients and controls across levels of the resting-state data and task hierarchy. Our results show that these functional network components as a function of task contain valuable information for individual prediction of schizophrenia patients. Such information is useful for training and replicates in testing. Performance was improved significantly (up to ~20%) relative to a single FNC (resting-state) measure.

Causal structure learning from time series data is a major scientific challenge. Existing algorithms assume that measurements occur sufficiently quickly; more precisely, they assume that the system and measurement timescales are approximately equal. In many scientific domains, however, measurements occur at a significantly slower rate than the underlying system changes. Moreover, the size of the mismatch between timescales is often unknown. This paper provides three distinct causal structure learning algorithms, all of which discover all dynamic graphs that could explain the observed measurement data as arising from undersampling at some rate. That is, these algorithms all learn causal structure without assuming any particular relation between the measurement and system timescales; they are thus rate-agnostic. We apply these algorithms to data from simulations. The results provide insight into the challenge of undersampling.

Time series shapelets are small subsequences that maximally differentiate classes of time series. Since the inception of shapelets, researchers have used shapelets for various data domains including anthropology and health care, and in the process suggested many efficient techniques for shapelet discovery. However, multi-dimensional time series data poses unique challenges to shapelet discovery that are yet to be solved. We show that an ensemble of shapelet-based decision trees on individual dimensions works better than shapelets defined over multiple dimensions. Generating a shapelet ensemble for multidimensional time series is computationally expensive. Most of the existing techniques prune shapelet candidates for speed. In this paper, we propose a novel technique for shapelet discovery that evaluates remaining candidates efficiently. Our algorithm uses a multi-length approximate index for time series data to efficiently find the nearest neighbors of the candidate shapelets. We employ a simple skipping technique for additional candidate pruning and a voting based technique to improve accuracy while retaining interpretability. Not only do we find a significant speed increase, our techniques enable us to efficiently discover shapelets on datasets with multi-dimensional and long time series such as hours of brain activity recordings. We demonstrate our approach on a biomedical dataset and find significant differences between patients with schizophrenia and healthy controls.

The joint independent component analysis (jICA) and the transposed independent vector analysis (tIVA) models are two effective solutions based on blind source separation (BSS) that enable fusion of data from multiple modalities in a symmetric and fully multivariate manner. The previous paper in this special issue discusses the properties and the main issues in the implementation of these two models. In this accompanying paper, we consider the application of these two models to fusion of multimodal medical imaging data-functional magnetic resonance imaging (fMRI), structural MRI (sMRI), and electroencephalography (EEG) data collected from a group of healthy controls and patients with schizophrenia performing an auditory oddball task. We show how both models can be used to identify a set of components that report on differences between the two groups, jointly, for all the modalities used in the study. We discuss the importance of algorithm and order selection as well as tradeoffs involved in the selection of one model over another. We note that for the selected data set, especially given the limited number of subjects available for the study, jICA provides a more desirable solution, however the use of an ICA algorithm that uses flexible density matching provides advantages over the most widely used algorithm, Infomax, for the problem.

Fusion of information from multiple sets of data in order to extract a set of features that are most useful and relevant for the given task is inherent to many problems we deal with today. Since, usually, very little is known about the actual interaction among the datasets, it is highly desirable to minimize the underlying assumptions. This has been the main reason for the growing importance of data-driven methods, and in particular of independent component analysis (ICA) as it provides useful decompositions with a simple generative model and using only the assumption of statistical independence. A recent extension of ICA, independent vector analysis (IVA) generalizes ICA to multiple datasets by exploiting the statistical dependence across the datasets, and hence, as we discuss in this paper, provides an attractive solution to fusion of data from multiple datasets along with ICA. In this paper, we focus on two multivariate solutions for multi-modal data fusion that let multiple modalities fully interact for the estimation of underlying features that jointly report on all modalities. One solution is the Joint ICA model that has found wide application in medical imaging, and the second one is the the Transposed IVA model introduced here as a generalization of an approach based on multi-set canonical correlation analysis. In the discussion, we emphasize the role of diversity in the decompositions achieved by these two models, present their properties and implementation details to enable the user make informed decisions on the selection of a model along with its associated parameters. Discussions are supported by simulation results to help highlight the main issues in the implementation of these methods.

We consider the problem of removing gradient artifact from electroencephalogram (EEG) signal, registered during a functional magnetic resonance imaging (fMRI) acquisition, by calculating and utilizing the statistical properties of the artifacts. We propose a new approach to EEG data organization for extracting artifactual components using independent vector analysis. This new approach estimates the gradient artifact signal as a single component thus alleviating the need of using advanced order selection algorithm before back reconstruction of EEG data. Experimental results are compared with average artifact subtraction method on real EEG data collected concurrently with fMRI data.

Independent component analysis (ICA) has proven quite useful for the analysis of functional magnetic resonance imaging (fMRI) data. However, stability of ICA decompositions is an issue in ICA of fMRI analysis primarily due to the noisy nature of fMRI data and the iterative nature of algorithms. In this work, we present an approach that utilizes an objective criterion and that is particularly suitable for image analysis to select the best of multiple ICA runs to use for further analysis and inference. In addition, a growing number of studies are focusing on the decomposition of single subject data and/or using high ICA model order, which both require an effective way to align components obtained from different ICA runs. In this paper, while presenting a method that provides superior performance in selecting the best run and interpreting the statistical reliability of ICA estimates, we also address the component sorting issue. Both simulated and real fMRI results show that our method selects more useful ICA runs than those selected by the widely used ICASSO software and that it is a more objective and better motivated approach to evaluate results and hence a promising tool for ICA analysis of fMRI data.

Independent component analysis (ICA) has been effectively used for the analysis of functional magnetic resonance imaging (fMRI) data, and recently for the analysis of fMRI data in its native complex-valued form. When performing complex ICA of fMRI analysis, it is desirable to take all three types of diversity - statistical property - that are present in the complex fMRI data into account: non-Gaussianity, sample dependence and noncircularity. In this paper, we study the performance of complex ICA by entropy rate bound minimization (CERBM) algorithm for fMRI analysis that takes all these three types of diversity into account. We perform a thorough comparison of its performance with that of complex Infomax (CInfomax) and complex ICA by entropy bound minimization (CEBM), two other important choices. While evaluating the performance of ICA algorithms for fMRI analysis, there are a number of challenges, including the lack of ground truth of fMRI data and inconsistent estimates due to the iterative nature of ICA algorithms. In this work, we also propose a statistical framework that utilizes an objective criterion to evaluate consistency of ICA algorithms. Using this framework, we show that CERBM leads to significant improvement in the estimations of components of interest in terms of providing lower mutual information rate, higher active scores and more numbers of activated voxels than those of CInfomax and CEBM, and the corresponding time courses present best task-relatedness with the paradigm.

Despite its multivariate nature, independent component analysis (ICA) is generally limited to univariate latents in the sense that each latent component is a scalar process. Independent subspace analysis (ISA), or multidimensional ICA (MICA), is a generalization of ICA which identifies latent independent vector components instead. While ISA/MICA considers multidimensional latent components within a single dataset, our work specifically considers the case of multiple datasets. Independent vector analysis (IVA) is a related technique that also considers multiple datasets explicitly but with a fixed and constrained model. Here, we first show that 1) ISA/MICA naturally extends to the case of multiple datasets (which we call MISA), and that 2) IVA is a special case of this extension. Then we develop an algorithm for MISA and demonstrate its performance on both IVA- and MISA-type problems. The benefit of these extensions is that the vector sources (or subspaces) capture higher order statistical dependence across datasets while retaining independence between subspaces. This is a promising model that can explore complex latent relations across multiple datasets and help identify novel biological traits for intricate mental illnesses such as schizophrenia.

We developed a new sparse multivariate regression method, collaborative sparse reduced rank regression(C-sRRR) for detecting genetic networks associated with brain functional networks in schizophrenia (SZ). Our study: 1) introduced both genetic and brain network structure to group single nucleotide polymorphism (SNP) and voxels simultaneously for utilizing the interacting effects implied in both features; 2) used collaborative sparse group lasso to perform genetic variants selection and nuclear norm penalty to address the interrelationship among voxels; 3) developed an efficient algorithm for solving the non-smooth optimization. In real data analysis, we constructed 8605 genetic sub-networks (modules) from 722177 SNPs with a median module size of 9. A functional brain network was extracted which also showed significant discriminative characteristics between SZ and healthy controls. A sub sampling strategy was applied to identify 57 highly ranked genes from 14 high-ranking modules. 14 of them are SZ susceptibility genes and 6 genes were consistent with the findings in previous study.

When applied to functional magnetic resonance imaging (fMRI) data, independent vector analysis (IVA) provides superior performance in capturing subject variability within one group, as compared to the widely used group independent component analysis (ICA) approach. However, the effectiveness of IVA algorithms in preserving variability between different groups of subjects has not been studied yet, although it is of great interest in most fMRI studies, especially for identifying biomarkers for diagnosis of mental disorders. In this paper, we introduce a methodology that uses graph-theoretical analysis and statistical analysis for assessing the ability of IVA algorithms to capture group variability. We generate multi-subject fMRI-like datasets with increasing spatial variability for a selected component between two groups and compare a robust IVA algorithm to group ICA approach. Our experimental results show that IVA can successfully preserve group variability, indicating its potential in extracting biomarkers across groups of subjects in fMRI analysis.

BACKGROUND: The emergence of high-throughput genomic datasets from different sources and platforms (e.g., gene expression, single nucleotide polymorphisms (SNP), and copy number variation (CNV)) has greatly enhanced our understandings of the interplay of these genomic factors as well as their influences on the complex diseases. It is challenging to explore the relationship between these different types of genomic data sets. In this paper, we focus on a multivariate statistical method, canonical correlation analysis (CCA) method for this problem. Conventional CCA method does not work effectively if the number of data samples is significantly less than that of biomarkers, which is a typical case for genomic data (e.g., SNPs). Sparse CCA (sCCA) methods were introduced to overcome such difficulty, mostly using penalizations with l-1 norm (CCA-l1) or the combination of l-1and l-2 norm (CCA-elastic net). However, they overlook the structural or group effect within genomic data in the analysis, which often exist and are important (e.g., SNPs spanning a gene interact and work together as a group). RESULTS: We propose a new group sparse CCA method (CCA-sparse group) along with an effective numerical algorithm to study the mutual relationship between two different types of genomic data (i.e., SNP and gene expression). We then extend the model to a more general formulation that can include the existing sCCA models. We apply the model to feature/variable selection from two data sets and compare our group sparse CCA method with existing sCCA methods on both simulation and two real datasets (human gliomas data and NCI60 data). We use a graphical representation of the samples with a pair of canonical variates to demonstrate the discriminating characteristic of the selected features. Pathway analysis is further performed for biological interpretation of those features. CONCLUSIONS: The CCA-sparse group method incorporates group effects of features into the correlation analysis while performs individual feature selection simultaneously. It outperforms the two sCCA methods (CCA-l1 and CCA-group) by identifying the correlated features with more true positives while controlling total discordance at a lower level on the simulated data, even if the group effect does not exist or there are irrelevant features grouped with true correlated features. Compared with our proposed CCA-group sparse models, CCA-l1 tends to select less true correlated features while CCA-group inclines to select more redundant features.

The nine papers in this special section represent the state-of-art in the area of multimodal biomedical imaging algorithms and applications.

It is important to identify significant biomarkers such as SNPs for medical diagnosis and treatment. However, the size of a biological sample is usually far less than the number of measurements, which makes the problem more challenging. To overcome this difficulty, we propose a sparse representation based variable selection (SRVS) approach. A simulated data set was first tested to demonstrate the advantages and properties of the proposed method. Then, we applied the algorithm to a joint analysis of 759075 SNPs and 153594 functional magnetic resonance imaging (fMRJ) voxels in 208 subjects (92 cases/116 controls) to identify significant biomarkers for schizophrenia (SZ). When compared with previous studies, our proposed method located 20 genes out of the top 45 SZ genes that are publicly reported We also detected some interesting functional brain regions from the fMRI study. In addition, a leave one out (LOO) cross-validation was performed and the results were compared with that of a previously reported method, which showed that our method gave significantly higher classification accuracy. In addition, the identification accuracy with integrative analysis is much better than that of using single type of data, suggesting that integrative analysis may lead to better diagnostic accuracy by combining complementary SNP and fMRI data.

We designed a sparse representation clustering (SRC) model to select the significant single nucleotide polymorphisms (SNPs) and proposed a novel SRC with a sliding window model for functional magnetic resonance imaging (fMRI) voxels selection. Then we combined two types of data to classify schizophrenia patients from healthy controls by linear support vector machine (SVM) to achieve a better diagnosis of schizophrenia. The effectiveness of the selected variables (SNPs or voxels) was validated by the leave one out (LOO) cross-validation method. The experimental results show that our proposed SRC method can effectively select the most discriminative variables in both SNPs and fMRI data. In particular, the combination of complementary fMRI and SNP data can significantly improve the classification of schizophrenia patients, which provides new insights in the study of schizophrenia.

In this paper we describe a deconvolution technique for obtaining an approximation of the neuronal signal from an observed hemodynamic response in fMRI data. Our approach, based on the Rauch-Tung-Striebel smoother for square-root cubature Kalman filter, enables us to accurately infer the hidden states, parameters, and the input of the dynamic system. Using a series of simulations we show in this paper that we are able to move beyond the limitation of a poorly sampled observation signal and estimate the true structure of underlying neuronal signal with significantly improved temporal resolution.

Nonnegative Least Squares (NNLS) is a general form for many important problems. We consider a special case of NNLS where the input is nonnegative. It is called Totally Nonnegative Least Squares (TNNLS) in the literature. We show a reduction of TNNLS to a single class Support Vector Machine (SVM), thus relating the sparsity of a TNNLS solution to the sparsity of supports in a SVM. This allows us to apply any SVM solver to the TNNLS problem. We get an order of magnitude improvement in running time by first obtaining a smaller version of our original problem with the same solution using a fast approximate SVM solver. Second, we use an exact NNLS solver to obtain the solution. We present experimental evidence that this approach improves the performance of state-of-the-art NNLS solvers by applying it to both randomly generated problems as well as to real datasets, calculating radiation therapy dosages for cancer patients.

DNA microarray gene expression and microarray-based comparative genomic hybridization (aCGH) have been widely used for biomedical discovery. Because of the large number of genes and the complex nature of biological networks, various analysis methods have been proposed. One such method is "gene shaving," a procedure which identifies subsets of the genes with coherent expression patterns and large variation across samples. Since combining genomic information from multiple sources can improve classification and prediction of diseases, in this paper we proposed a new method, "ICA gene shaving" (ICA, independent component analysis), for jointly analyzing gene expression and copy number data. First we used ICA to analyze joint measurements, gene expression and copy number, of a biological system and project the data onto statistically independent biological processes. Next, we used these results to identify patterns of variation in the data and then applied an iterative shaving method. We investigated the properties of our proposed method by analyzing both simulated and real data. We demonstrated that the robustness of our method to noise using simulated data. Using breast cancer data, we showed that our method is superior to the Generalized Singular Value Decomposition (GSVD) gene shaving method for identifying genes associated with breast cancer.

We propose a data fusion method for the fusion of simultaneously acquired functional magnetic resonance imaging (fMRI) and single trial electroencephalography (EEG) data from multiple subjects using multi-set canonical correlation analysis (M-CCA). Our proposed technique utilizes the common time series information in the multimodal datasets to find trial-to-trial covariations across modalities, and based on these covariations, the data is decomposed into spatial maps for the fMRI data and a corresponding temporal evolution for the EEG data. Additionally, the analysis is performed simultaneously on data from a group of subjects, thus providing an efficient tool to make group inferences about cross-modality covariation. The proposed method is multivariate and hence facilitates the study of brain connectivity along with localization of brain function. We demonstrate the promise of the method in finding covarying trial-to-trial amplitude modulations in an auditory task involving implicit pattern learning.

Data-driven analysis methods, in particular independent component analysis (ICA) has proven quite useful for the analysis of functional magnetic imaging (fMRI) data. In addition, by enabling one to work in its native, complex form, complex-valued ICA algorithms provide better estimation performance compared to the traditional approach that uses only the magnitude data. In the complex domain, circularity has been a common assumption even though most data acquisition methods collect fMRI data that end up being noncircular when saved in complex form. In this paper, we show that a complex ICA approach that does not assume circularity and also adapts to the source density is the more desirable one for performing ICA of complex fMRI data. We show that by adaptively matching the underlying fMRI density model, the analysis performance can be improved in terms of both the estimation of the task-related time courses and in the spatial activation.

Independent component analysis (ICA) has been successfully applied for the analysis of functional magnetic resonance imaging (fMRI) data. However, independence might be too strong a constraint for certain sources. In this paper, we present an independent subspace analysis (ISA) framework that forms independent subspaces among the estimated sources having dependencies by a hierarchial clustering approach and subsequently separates the dependent sources in the task-related subspace using prior information. We study the incorporation of two types of prior information to transform the sources within the task-related subspace: sparsity and task-related time courses. We demonstrate the effectiveness of our proposed method for source separation of multi-subject fMRI data from a visuomotor task. Our results show that physiologically meaningful dependencies among sources can be identified using our subspace approach and the dependent estimated components can be further separated effectively using a subsequent transformation.

Analysis of functional magnetic resonance imaging (fMRI) data in its native, complex form has been shown to increase the sensitivity of the analysis both for data driven techniques such as independent component analysis (ICA) and for model-driven techniques; however, the noisy nature of the phase poses a challenge for successful study of fMRI data. In addition, for complex ICA, the inherent scaling ambiguity, which has a phase term, introduces additional difficulty for group analysis and visualization of the results. In this paper, we address these issues, which have been among the main reasons phase information has been traditionally discarded and introduce a phase correction scheme that can be either applied subsequent to ICA of fMRI data or can be incorporated into the ICA algorithm in the form of prior information to eliminate the need for further processing for phase correction. In addition, we introduce methods for visualization of the analysis results as well as preprocessing the complex fMRI data to mitigate the effects of noise in the phase which are not limited to ICA algorithms. We demonstrate the successful application of the methods using actual fMRI data.

Distributions over permutations arise in applications ranging from multi-object tracking to ranking of instances. The difficulty of dealing with these distributions is caused by the size of their domain, which is factorial in the number of considered entities (n!). It makes the direct definition of a multinomial distribution over permutation space impractical for all but a very small n. In this work we propose an embedding of all n! permutations for a given n in a surface of a hyper sphere defined in ℝ(n-1). As a result of the embedding, we acquire ability to define continuous distributions over a hyper sphere with all the benefits of directional statistics. We provide polynomial time projections between the continuous hyper sphere representation and the n!-element permutation space. The framework provides a way to use continuous directional probability densities and the methods developed thereof for establishing densities over permutations. As a demonstration of the benefits of the framework we derive an inference procedure for a state-space model over permutations. We demonstrate the approach with simulations on a large number of objects hardly manageable by the state of the art inference methods, and an application to a real flight traffic control dataset.

Typically data acquired through imaging techniques such as functional magnetic resonance imaging (fMRI), structural MRI (sMRI), and electroencephalography(EEG) are analyzed separately. Each modality records brain structure and function at different scales, and fusing information from such complementary modalities promises to provide additional insight into connectivity across brain networks and changes due to disease. Recently, a number of methods have been proposed for data integration and fusion of two brain imaging modalities. We propose a new data fusion scheme based on canonical correlation analysis that enables the detection of associations across multiple modalities. Our multimodal canonical correlation analysis (mCCA) scheme works at the feature level using multi-set CCA to determine inter-subject covariations across modalities. We apply mCCA to fMRI, sMRI, and EEG data collected from patients diagnosed with schizophrenia and healthy controls. Through data collected from an auditory oddball task, we show that the fusion of multiple modalities detects more specific associations as compared to fusion of two modalities.

The dual formulation of the support vector machine (SVM) objective function is an instance of a nonnegative quadratic programming problem. We reformulate the SVM objective function as a matrix factorization problem which establishes a connection with the regularized nonnegative matrix factorization (NMF) problem. This allows us to derive a novel multiplicative algorithm for solving hard and soft margin SVM. The algorithm follows as a natural extension of the updates for NMF and semi-NMF. No additional parameter setting, such as choosing learning rate, is required. Exploiting the connection between SVM and NMF formulation, we show how NMF algorithms can be applied to the SVM problem. Multiplicative updates that we derive for SVM problem also represent novel updates for semi-NMF. Further this unified view yields algorithmic insights in both directions: we demonstrate that the Kernel Adatron algorithm for solving SVMs can be adapted to NMF problems. Experiments demonstrate rapid convergence to good classifiers. We analyze the rates of asymptotic convergence of the updates and establish tight bounds. We test them on several datasets using various kernels and report equivalent classification performance to that of a standard SVM.

The acquisition of multiple brain imaging types for a given study is a very common practice. There have been a number of approaches proposed for combining or fusing multitask or multimodal information. These can be roughly divided into those that attempt to study convergence of multimodal imaging, for example, how function and structure are related in the same region of the brain, and those that attempt to study the complementary nature of modalities, for example, utilizing temporal EEG information and spatial functional magnetic resonance imaging information. Within each of these categories, one can attempt data integration (the use of one imaging modality to improve the results of another) or true data fusion (in which multiple modalities are utilized to inform one another). We review both approaches and present a recent computational approach that first preprocesses the data to compute features of interest. The features are then analyzed in a multivariate manner using independent component analysis. We describe the approach in detail and provide examples of how it has been used for different fusion tasks. We also propose a method for selecting which combination of modalities provides the greatest value in discriminating groups. Finally, we summarize and describe future research topics.

In this work, we propose a scheme for joint blind source separation (BSS) of multiple datasets using canonical correlation analysis (CCA). The proposed scheme jointly extracts sources from each dataset in the order of between-set source correlations. We show that, when sources are uncorrelated within each dataset and correlated across different datasets only on corresponding indices, (i) CCA on two datasets achieves BSS when the sources from the two datasets have distinct between-set correlation coefficients, and (ii) CCA on multiple datasets (M-CCA) achieves BSS with a more relaxed condition on the between-set source correlation coefficients compared to CCA on two datasets. We present simulation results to demonstrate the properties of CCA and M-CCA on joint BSS. We apply M-CCA to group functional magnetic resonance imaging (fMRI) data acquired from several subjects performing a visuomotor task and obtain interesting brain activations as well as their correlation profiles across different subjects in the group.

This paper presents an approach for selecting optimal components for discriminant analysis. Such an approach is useful when further detailed analyses for discrimination or characterization requires dimensionality reduction. Our approach can accommodate a categorical variable such as diagnosis (e.g. schizophrenic patient or healthy control), or a continuous variable like severity of the disorder. This information is utilized as a reference for measuring a component's discriminant power after principle component decomposition. After sorting each component according to its discriminant power, we extract the best components for discriminant analysis. An application of our reference selection approach is shown using a functional magnetic resonance imaging data set in which the sample size is much less than the dimensionality. The results show that the reference selection approach provides an improved discriminant component set as compared to other approaches. Our approach is general and provides a solid foundation for further discrimination and classification studies.

Independent component analysis (ICA) is a statistical and computational technique for revealing hidden factors that underlie sets of signals. We propose an improved ICA framework for group data analysis by adding an adaptive constraint to the mixing coefficients, namely, constrained coefficients ICA (CCICA). The method is dedicated to identification and increasing the accuracy of components that show significant group differences reflected in the mixing coefficients. Performance of CCICA is assessed by simulations under different signal to noise ratios. An application to multitask functional magnetic resonance imaging analysis is conducted to illustrate the advantages of CCICA. It is shown that CCICA provides stable results and can estimate both the components and the mixing coefficients with a relatively high accuracy compared to Infomax, hence is a promising tool for the identification of biomarkers from brain imaging data.

Functional magnetic resonance imaging (fMRI) data are originally acquired as complex-valued images, while virtually all fMRI studies only use the magnitude of the data in the analysis. Since little is known for devising models for the phase, independent component analysis (ICA) emerges as a promising technique for data-driven analysis of fMRI data in its native complex form. In this paper, we compare the performance of ICA on real-valued and complex-valued fMRI data and show the advantages of the complex approach. We also develop complex-valued order selection scheme to improve the estimation of the number of independent components in complex-valued fMRI data using information-theoretic criteria. Comparisons on order selection using real-valued and complex-valued fMRI data demonstrate the more informative nature of complex data.

We present a novel multivariate approach called source based morphometry (SBM) to study the novel structural MRI phase images and get sources of relative gray matter and white matter differences between patients and healthy controls. SBM considers the information cross brain voxels and provides spatially maximal independent sources about localization of changes. The structural MRI phase images efficiently summarize the relationship that exists between the gray and white matter without having to increase the dimensionality of the problem. SBM was then applied to the phase images. Results identified patient versus control differences in gray matter and white matter for visual-motor cortex as well as other areas. These interesting findings show that SBM is a useful multivariate approach for studying the brain. Moreover, the use of structural MRI phase images to joint gray and white matter together provides a significant advantage.

Non-negative matrix factorization (NMF) has increasingly been used as a tool in signal processing in the last couple of years. NMF, like independent component analysis (ICA) is useful for decomposing high dimensional data sets into a lower dimensional space. Here, we use NMF to learn the features of both structural and functional magnetic resonance imaging (sMRI/fMRI) data. NMF can be applied to perform group analysis of imaging data and we apply it to learn the spatial patterns which linearly covary among subjects for both sMRI and fMRI. We add an additional contrast term to NMF (called co-NMF) to identify features distinctive between two groups. We apply our approach to a dataset consisting of schizophrenia patients and healthy controls. The results from co-NMF make sense in light of expectations and are improved compared to the NMF results. Our method is general and may prove to be a useful tool for identifying differences between multiple groups.

Relationships between genomic data and functional brain images are of great interest but require new analysis approaches to integrate the high-dimensional data types. This letter presents an extension of a technique called parallel independent component analysis (paraICA), which enables the joint analysis of multiple modalities including interconnections between them. We extend our earlier work by allowing for multiple interconnections and by providing important overfitting controls. Performance was assessed by simulations under different conditions, and indicated reliable results can be extracted by properly balancing overfitting and underfitting. An application to functional magnetic resonance images and single nucleotide polymorphism array produced interesting findings.

Functional magnetic resonance imaging (fMRI) data provides spatially localized subcentimeter information about blood flow and oxygenation secondary to neuronal activation, but with temporal resolution on the order of seconds. Event-related potential (ERP) studies provide millimeter resolution measurements of the electric changes induced by neuronal activity, but spatial information is not well localized and suffers from an ill-posed inverse problem since there are much fewer sensors than solutions. Combining or fusing these two techniques thus has the potential to provide simultaneous higher temporal and high spatial resolution. Localization of the brain's response to infrequent, task-relevant target 'oddball' stimuli in humans has remained challenging due to the lack of a single imaging technique with good spatial and temporal resolution. In this paper, we use independent component analysis to fuse ERP and fMRI modalities to identify, for the first time in humans, the dynamics of the auditory oddball response with high spatiotemporal resolution across the entire brain. The results illuminate a new era of brain research utilizing the precise temporal information in ERPs and the high spatial resolution of fMRI

We study the performance of five blind source separation (BSS) algorithms when applied to analysis of functional magnetic resonance imaging (fMRI) data. We introduce a Matlab-based toolbox, the group ICA of fMRI toolbox (GIFT), which enables analysis of groups of subjects using BSS algorithms, in particular those based on independent component analysis (ICA). We use the visualization and computational tools included in GIFT to quantitatively analyze the performance of different BSS algorithms for fMRI analysis and discuss the results.

We present a projection-based framework for a feature-selective independent component analysis (FS-ICA) scheme and study its convergence property for two ICA algorithms, FastICA and Infomax. As examples, we implement bandpass filter as the feature-selective filter to improve the estimation of a bandpass signal from the mixtures and a periodic task-related time course embedded in the functional magnetic resonance imaging (fMRI) data. Hence, we demonstrate that the proposed method can incorporate a priori information into ICA to effectively improve estimation of the underlying components of practical interest, such as periodic time courses and smooth brain activation areas in fMRI data.

This paper deals with the problem of blind source separation in fMRI data analysis. Our main contribution is to present a maximum likelihood based method to blindly separate the brain activations in an fMRI experiment. Choosing the time frequency domain as the signal representation space, our method relies on the second order statistics and exploits the inter-source diversity. It is efficiently implemented by the EM (expectation-maximization) algorithm where the time courses of the brain activations are considered as the hidden variables. The estimation variance of the STFT (short time Fourier transform) is reduced by averaging across time frequency sub-domains. The successful separation of the right and left visual cortex activations during a visual fMRI experiment, in a block design, and the extraction of only the relevant tasks corroborate the effectiveness of our proposed separating algorithm.

A number of complex nonlinear functions are proposed for the independent component analysis (ICA) of complex-valued data. We discuss the properties of these nonlinearities and show their efficiency in generating the higher order statistics needed for ICA.

Independent component analysis (ICA) for separating complex-valued sources is needed for convolutive source-separation in the frequency domain, or for performing source separation on complex-valued data, such as functional magnetic resonance imaging data. Functional magnetic resonance imaging (fMRI) is a technique that produces complex-valued data; however the vast majority of fMRI analyses utilize only magnitude images. We compare the performance of the complex infomax. algorithm that uses an analytic (and hence unbounded) nonlinearity with the traditional complex infomax approaches that employ bounded (and hence non-analytic) nonlinearities as well as with a cumulant-based approach. We compare the performances of these algorithms for processing both simulated and real fMRI data and show that the complex infomax. using analytic nonlinearity has the ability to separate both sub- and super-Gaussian sources with a hyperbolic tangent nonlinearity. The complex infomax algorithm that uses analytic nonlinearity thus provides a potentially powerful method for exploratory analysis of fMRI data.

Functional magnetic resonance imaging (fMRI) is a technique which produces complex data; however the vast majority of functional magnetic resonance imaging analyses utilize only magnitude images. In this paper, we derive a complex-valued independent component analysis (ICA) algorithm using the infomax approach which we then apply to fMRI analysis. Theoretical and empirical results demonstrate an improved sensitivity to functional changes when utilizing the complex data. Additionally, the complex infomax algorithm developed provides a powerful method for exploratory analysis of fMRI data.