Methods based on independent component analysis (ICA) and canonical correlation analysis (CCA) as well as their various extensions have become popular for the fusion of multimodal data as they minimize assumptions about the relationships among multiple datasets. Two important extensions that are widely used, joint ICA (jICA) and parallel ICA (pICA), make a number of simplifying assumptions that might limit their usefulness such as identical mixing matrices for jICA, and the requirement for the same number of components for jICA and pICA. In this paper, we propose a new, flexible hybrid method for fusion based on ICA and CCA, called consecutive independence and correlation transform (C-ICT), which relaxes the main limitations of jICA and pICA. We demonstrate performance advantages of C-ICT both through simulations and application to real medical data collected from schizophrenia patients and healthy controls performing an auditory oddball task (AOD).

Dynamic functional connectivity has become a prominent approach for tracking the changes of macroscale statistical dependencies between regions in the brain. Effective parametrization of these statistical dependencies, referred to as brain states, is however still an open problem. We investigate different emission models in the hidden Markov model framework, each representing certain assumptions about dynamic changes in the brain. We evaluate each model by how well they can discriminate between schizophrenic patients and healthy controls based on a group independent component analysis of resting-state functional magnetic resonance imaging data. We find that simple emission models without full covariance matrices can achieve similar classification results as the models with more parameters. This raises questions about the predictability of dynamic functional connectivity in comparison to simpler dynamic features when used as biomarkers. However, we must stress that there is a distinction between characterization and classification, which has to be investigated further.

Due to the potential for object occlusion in crowded areas, the use of multiple cameras for video surveillance has prevailed over the use of a single camera. This has motivated the development of a number of techniques to analyze such multi-camera video sequences. However, most of these techniques require a camera calibration step, which is cumbersome and must be done for every new configuration. Additionally, these techniques fail to exploit the complementary information across these multiple datasets. We propose a data-driven solution to the problem by making use of the inherent similarity of temporal signatures of objects across video sequences. We introduce an effective solution for the detection of abandoned objects using this inherent diversity based on the transposed independent vector analysis (tIVA) model. By taking advantage of the similarity across multiple cameras, the new technique does not require any calibration and thus can be readily applied to any camera configuration. We demonstrate the superior performance of our technique over the single camera-based method using the PETS 2006 dataset.

The existence of complementary information across multiple sensors has driven the proliferation of multivariate datasets. Exploitation of this common information, while minimizing the assumptions imposed on the data has led to the popularity of data-driven methods. Independent vector analysis (IVA), in particular, provides a flexible and effective approach for the fusion of multivariate data. In many practical applications, important prior information about the data exists and incorporating this information into the IVA model is expected to yield improved separation performance. In this paper, we propose a general formulation for non-orthogonal constrained IVA (C-IVA) framework that can incorporate prior information about either the sources or the mixing coefficients into the IVA cost function. A powerful decoupling method is the major enabling factor in this task. We demonstrate the improved performance of C-IVA over the unconstrained IVA model using both simulated as well as real medical imaging data.

Independent component analysis (ICA) is a powerful method for blind source separation based on the assumption that sources are statistically independent. Though ICA has proven useful and has been employed in many applications, complete statistical independence can be too restrictive an assumption in practice. Additionally, important prior information about the data, such as sparsity, is usually available. Sparsity is a natural property of the data, a form of diversity, which, if incorporated into the ICA model, can relax the independence assumption, resulting in an improvement in the overall separation performance. In this work, we propose a new variant of ICA by entropy bound minimization (ICA-EBM)-a flexible, yet parameter-free algorithm-through the direct exploitation of sparsity. Using this new SparseICA-EBM algorithm, we study the synergy of independence and sparsity through simulations on synthetic as well as functional magnetic resonance imaging (fMRI)-like data.

Exploitation of complementary information is the principal reason for collecting data from multiple neurological sensors. Since little is known about the latent processes underlying neural function, it is important to minimize the assumptions placed on the data when performing a joint analysis. This motivates the use of data-driven fusion methods, such as independent vector analysis (IVA), for the analysis of neurological data. For neural datasets, the complementary information exploited by fusion methods may be in the form of similar spatial activation across datasets, the spatial IVA (sIVA) model, or similar subject relations across datasets, the transposed IVA (tIVA) model. Despite the potential power of these two models, no study has investigated how the differences in the modeling assumptions of sIVA and tIVA inform the fusion of real neuro-imaging data. In this paper, we utilize a unique set of multitask functional magnetic resonance imaging data from 271 subjects to directly compare the sIVA and tIVA models and visualize their differences using a novel technique, global difference maps. Through this application, we note important similarities between the results from the two methods that increase our confidence in their overall performance, though differences in modeling assumptions result in certain differences in the decompositions.

The use of in vivo calcium imaging has granted researchers the unprecedented ability to study large populations of neurons in real time, enabling direct observation of how the brain processes information. Such data offers great potential, however for current analysis techniques, successful extraction of the true neuronal signals is intimately tied to the proper selection of multiple user-defined parameters, which must be tuned for each video sequence. To overcome such issues, we propose a novel parameter-free independent component analysis (ICA)-based method, ICA with signal reconstruction and ordering (ICA+SRO), to automatically extract neuronal signals from calcium imaging sequences. The power of ICA+SRO is demonstrated on a real calcium imaging sequence. We compare the results of ICA+SRO with those from the popular principal component analysis-ICA based technique and show significant improvement. The results demonstrate the simplicity of a parameter-free method and its power in extracting neuronal signals from calcium imaging sequences.

Neuroimaging modalities such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide information about neurological functions in complementary spatiotemporal resolutions; therefore, fusion of these modalities is expected to provide better understanding of brain activity. In this paper, we jointly analyze fMRI and EEG data collected during an auditory oddball task with the goal of capturing brain activity patterns that differ between patients with schizophrenia and healthy controls. Rather than selecting a single electrode or matricizing the third-order tensor that can be naturally used to represent multi-channel EEG signals, we preserve the multi-way structure of EEG data and use a coupled matrix and tensor factorization (CMTF) model to jointly analyze fMRI and EEG signals. Our analysis reveals that (i) joint analysis of EEG and fMRI using a CMTF model can capture meaningful temporal and spatial signatures of patterns that behave differently in patients and controls, and (ii) these differences and the interpretability of the associated components increase by including multiple electrodes from frontal, motor and parietal areas, but not necessarily by including all electrodes in the analysis.

The extraction of information from multiple sets of data is a problem inherent to many disciplines. This is possible by either analyzing the data sets jointly as in data fusion or separately and then combining as in data integration. However, selecting the optimal method to combine and analyze multiset data is an ever-present challenge. The primary reason for this is the difficulty in determining the optimal contribution of each data set to an analysis as well as the amount of potentially exploitable complementary information among data sets. In this paper, we propose a novel classification rate-based technique to unambiguously quantify the contribution of each data set to a fusion result as well as facilitate direct comparisons of fusion methods on real data and apply a new method, independent vector analysis (IVA), to multiset fusion. This classification rate-based technique is used on functional magnetic resonance imaging data collected from 121 patients with schizophrenia and 150 healthy controls during the performance of three tasks. Through this application, we find that though optimal performance is achieved by exploiting all tasks, each task does not contribute equally to the result and this framework enables effective quantification of the value added by each task. Our results also demonstrate that data fusion methods are more powerful than data integration methods, with the former achieving a classification rate of 73.5 % and the latter achieving one of 70.9 %, a difference which we show is significant when all three tasks are analyzed together. Finally, we show that IVA, due to its flexibility, has equivalent or superior performance compared with the popular data fusion method, joint independent component analysis.

Automated detection of abandoned object (AO) is an important application in video surveillance for security purposes. Because of its importance, a number of techniques have been proposed to automatically detect abandoned objects in the past years. However, these techniques require prior knowledge on the properties of the object such as its shape and color, in order to classify foreground objects as abandoned object. In contrast, independent component analysis (ICA) does not require such prior knowledge. However, it can only model one dataset at a time, thus limiting its usage to monochrome frames. In this paper, we propose to use independent vector analysis (IVA), a recent extension of ICA to multivariate data that takes the dependence across multiple datasets into account while retaining the independence within each dataset. We present a new framework for AO detection using IVA and show that it provides successful performance in complicated scenarios, such as for videos with crowd, illumination change, and occlusion.

The joint independent component analysis (jICA) and the transposed independent vector analysis (tIVA) models are two effective solutions based on blind source separation (BSS) that enable fusion of data from multiple modalities in a symmetric and fully multivariate manner. The previous paper in this special issue discusses the properties and the main issues in the implementation of these two models. In this accompanying paper, we consider the application of these two models to fusion of multimodal medical imaging data-functional magnetic resonance imaging (fMRI), structural MRI (sMRI), and electroencephalography (EEG) data collected from a group of healthy controls and patients with schizophrenia performing an auditory oddball task. We show how both models can be used to identify a set of components that report on differences between the two groups, jointly, for all the modalities used in the study. We discuss the importance of algorithm and order selection as well as tradeoffs involved in the selection of one model over another. We note that for the selected data set, especially given the limited number of subjects available for the study, jICA provides a more desirable solution, however the use of an ICA algorithm that uses flexible density matching provides advantages over the most widely used algorithm, Infomax, for the problem.

Fusion of information from multiple sets of data in order to extract a set of features that are most useful and relevant for the given task is inherent to many problems we deal with today. Since, usually, very little is known about the actual interaction among the datasets, it is highly desirable to minimize the underlying assumptions. This has been the main reason for the growing importance of data-driven methods, and in particular of independent component analysis (ICA) as it provides useful decompositions with a simple generative model and using only the assumption of statistical independence. A recent extension of ICA, independent vector analysis (IVA) generalizes ICA to multiple datasets by exploiting the statistical dependence across the datasets, and hence, as we discuss in this paper, provides an attractive solution to fusion of data from multiple datasets along with ICA. In this paper, we focus on two multivariate solutions for multi-modal data fusion that let multiple modalities fully interact for the estimation of underlying features that jointly report on all modalities. One solution is the Joint ICA model that has found wide application in medical imaging, and the second one is the the Transposed IVA model introduced here as a generalization of an approach based on multi-set canonical correlation analysis. In the discussion, we emphasize the role of diversity in the decompositions achieved by these two models, present their properties and implementation details to enable the user make informed decisions on the selection of a model along with its associated parameters. Discussions are supported by simulation results to help highlight the main issues in the implementation of these methods.