Recent progress in artificial intelligence has encouraged numerous attempts to understand and decode human visual system from brain signals. These prior works typically align neural activity independently with semantic and perceptual features extracted from images using pre-trained vision models. However, they fail to account for two key challenges: (1) the modality gap arising from the natural difference in the information level of representation between brain signals and images, and (2) the fact that semantic and perceptual features are highly entangled within neural activity. To address these issues, we utilize hyperbolic space, which is well-suited for considering differences in the amount of information and has the geometric property that geodesics between two points naturally bend toward the origin, where the representational capacity is lower. Leveraging these properties, we propose a novel framework, Hyperbolic Feature Interpolation (HyFI), which interpolates between semantic and perceptual visual features along hyperbolic geodesics. This enables both the fusion and compression of perceptual and semantic information, effectively reflecting the limited expressiveness of brain signals and the entangled nature of these features. As a result, it facilitates better alignment between brain and visual features. We demonstrate that HyFI achieves state-of-the-art performance in zero-shot brain-to-image retrieval, outperforming prior methods with Top-1 accuracy improvements of up to +17.3% on THINGS-EEG and +9.1% on THINGS-MEG.

Resting-state functional magnetic resonance imaging (rs-fMRI) measures intrinsic neural activity, and analyzing its frequency-domain characteristics provides insights into brain dynamics. Owing to these properties, rs-fMRI is widely used to investigate brain disorders such as autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD). Conventional frequency-domain analyses typically rely on the Fourier transform, which lacks flexibility in capturing non-stationary neural signals due to its fixed resolution. Furthermore, these methods primarily utilize only real-valued features, such as the magnitude or phase, derived from complex-valued spectral representations. Consequently, direct modeling of the real and imaginary components, particularly within fMRI analyses, remains largely unexplored, overlooking the distinct and complementary spectral information encoded in these components. To address these limitations, we propose a novel Transformer-based framework that explicitly models the real and imaginary components of continuous wavelet transform (CWT) coefficients from rs-fMRI signals. Our architecture integrates spectral, temporal, and spatial attention modules, employing self- and cross-attention mechanisms to jointly capture intra- and inter-component relationships. Applied to the Autism Brain Imaging Data Exchange (ABIDE)-I and ADHD-200 datasets, our approach achieved state-of-the-art classification performance compared to existing baselines. Comprehensive ablation studies demonstrated the advantages of directly utilizing real and imaginary components over conventional frequency-domain features and validate each module's contribution. Moreover, attention-based analyses revealed frequency- and region-specific patterns consistent with known neurobiological alterations in ASD and ADHD. These findings highlight that preserving and jointly leveraging the real and imaginary components of CWT-based representations not only enhances diagnostic performance but also provides interpretable insights into neurodevelopmental disorders.

Ultrahigh-field (UHF) magnetic resonance imaging (MRI), 7T MRI, provides superior anatomical details of internal brain structures thanks to its enhanced signal-to-noise ratio and susceptibility-induced contrast. However, the widespread use of 7T MRI is limited by its high cost and lower accessibility compared to low-field (LF) MRI. This study proposes a SegUHF that systematically fuses the input LF MRI feature representations with the inferred 7T-like feature representations for brain image segmentation tasks in a 7T-absent environment. Specifically, our proposed adaptive fusion module within the SegUHF aggregates 7T-like features derived from the LF image using a pre-trained network and then refines them to effectively assimilate UHF guidance into LF image features. Using intensity-guided features obtained from such aggregation and assimilation, segmentation models can recognize subtle structural representations that are usually difficult to identify when relying only on LF features. Beyond these advantages, this strategy can be seamlessly utilized by modulating the contrast of LF features in alignment with UHF guidance, even when employing arbitrary segmentation models. Extensive experiments demonstrated that our method outperformed all baselines in both brain tissue and whole-brain segmentation, while also showcasing adaptability and scalability across various models and tasks. Code is available at https://github.com/ku-milab/UHF-guided_segmentation

Accurate identification of brain disorders enables timely intervention and improved patient outcomes. While numerous studies have developed AI models for resting-state functional magnetic resonance imaging (rs-fMRI) analysis, most rely on supervised learning, which can overlook hidden patterns that are less discriminatively associated with labels and require large annotated datasets. To address these limitations, we propose leveraging normative modeling, an unsupervised approach that constructs a model of normality based on healthy controls' data. Deviations from normality indicate potential disorders. However, applying normative modeling to rs-fMRI faces two significant challenges: constructing normality and ensuring explainability. To tackle these challenges, we propose BRAINEXA, a novel framework enhancing normative modeling for rs-fMRI-based brain disorder identification. Specifically, to construct accurate and stable normality, BRAINEXA introduces a training strategy that predicts more informative regions from less informative regions, discouraging trivial self-supervised learning solutions and improving representation learning without additional overhead. Furthermore, we incorporate spatiotemporal mutual information regularization to preserve distinctiveness between more informative regions and less informative regions during latent encoding, preventing potential representational distortions. For interpretability, BRAINEXA extracts normality-defining (ND) subregions, the core regions that characterize normal brain function. By combining ND subregions with anomaly scores, BRAINEXA can offer region- and connection-wise explanations that help identify clinically meaningful disruptions of normality in an unsupervised setting. We demonstrate the effectiveness of BRAINEXA on four public rs-fMRI datasets: REST-meta-MDD, ABIDE I, ADHD-200, and OASIS-3. Our code is available at https://github.com/ku-milab/BRAINEXA.

Single-source domain generalization (SDG) in medical image segmentation (MIS) aims to generalize a model using only one source domain data to segment data from an unseen target domain. Despite substantial advances in SDG with data augmentation, existing methods often fail to fully consider the details and uncertain areas prevalent in MIS, leading to mis-segmentation. In this study, we propose a Fourier-based semantic augmentation method called FIESTA using uncertainty guidance (UG) to enhance the fundamental goals of MIS in an SDG context by manipulating the amplitude and phase components in the frequency domain. The proposed Fourier augmentative transformer (FAT) addresses semantic amplitude modulation based on meaningful angular points to induce pertinent variations and harnesses the phase spectrum to ensure structural coherence. Moreover, FIESTA employs uncertainty estimation to fine-tune the augmentation process, improving the ability of the model to adapt to diverse augmented data and concentrate on areas with higher ambiguity. Extensive experiments across three cross-domain scenarios demonstrate that FIESTA surpasses recent state-of-the-art SDG approaches in segmentation performance and significantly contributes to boosting the model's applicability in medical imaging modalities.

Electroencephalogram (EEG)-based brain-computer interfaces face challenges in data insufficiency to train neural networks for tasks and generalizability for multiple subjects. To address the issue of data scarcity in EEG research, transfer learning using EEG foundation models (EFMs) has recently gained attention for its ability to leverage prior knowledge. Although transfer learning with EFMs enables tasks to be performed with limited training data, their increasing size presents significant computational challenges. Parameter-efficient transfer learning (PETL) methods address this computational issue by tuning only a small subset of parameters from the pre-trained model. However, existing PETL methods mostly fail to account for the high-dimensional nature of EEG data, which limits their ability to fully leverage the prior knowledge of the EFM when applied to downstream tasks. To address these challenges, we propose a novel PETL method with a TASk-relevanT fEature Focusing modULe (TASTEFUL) to transfer EFMs efficiently. TASTEFUL is designed to focus on task-relevant features and efficiently learn representations tailored for downstream tasks. We evaluated our proposed TASTEFUL on tasks using publicly available EEG datasets, demonstrating its superior performance. Finally, our work highlights TASTEFUL's potential to enhance the practical application of EFMs, marking a significant advancement in PETL for EFMs.

Time series data, widely used in fields such as climate studies, finance, and healthcare, often face scarcity in rare scenarios and privacy concerns, prompting growing interest in time series synthesis. Diffusion models have shown strong potential for generating high-quality data, but challenges remain in capturing long-range dependencies and complex patterns. We propose a novel diffusion model that integrates time-domain information with rich frequency-domain features, accounting for differences in noise decay rates across frequencies. Instead of arbitrary frequency splits used in prior works, we partition components based on spectral density, model them separately within the denoising backbone, and fuse them with time-domain features. This enables effective capture of both global and local patterns, enhancing representation of high- and low-frequency information. Extensive experiments on multiple public datasets show promising performance, and analyses including long-term generation and ablation studies demonstrate the model's ability to learn and represent complex time series distributions.

Causal inference involves determining how interventions affect outcomes and explaining the underlying mechanisms, and it holds critical importance across various fields. A key assumption in causal inference is that the measured covariates form a sufficient adjustment set. However, this assumption often fails due to unobserved confounders, as confounding mechanisms are rarely fully captured by measured covariates alone. Recent research has attempted to address this challenge using variational autoencoders (VAEs), but these approaches face practical limitations, including unidentifiability and bias toward proxy variables. To overcome these issues, we propose a novel method that incorporates quantized factor identifiability into VAEs for causal effect estimation. This integration mitigates unidentifiability and reduces the dominance of proxy variables, thereby enhancing consistency and accuracy in causal inference. Extensive experiments on both simulated and real-world datasets demonstrate the robustness and effectiveness of our method, establishing a new benchmark in deep causal modeling.

Vision-language models have achieved remarkable performance across various tasks by leveraging large-scale multimodal training data. However, their ability to generalize to out-of-distribution (OOD) domains requiring expert-level knowledge remains an open challenge. To address this, we investigate cross-domain transfer learning approaches for efficiently adapting diffusion classifiers to new target domains demanding expert-level domain knowledge. Specifically, we propose ExpertDiff, a head-less model reprogramming technique that optimizes the instruction-following abilities of text-to-image diffusion models via learnable prompts, while leveraging the diffusion classifier objective as a modular plug-and-play adaptor. Our approach eliminates the need for conventional output mapping layers (e.g., linear probes), enabling seamless integration with off-the-shelf diffusion frameworks like Stable Diffusion. We demonstrate the effectiveness of ExpertDiff on the various OOD datasets (i.e., medical and satellite imagery). Furthermore, we qualitatively showcase ExpertDiff’s ability to faithfully reconstruct input images, highlighting its potential for both downstream discriminative and upstream generative tasks. Our work paves the way for effectively repurposing powerful foundation models for novel OOD applications requiring domain expertise.

Dynamic treatment regimes (DTRs), which comprise a series of decisions taken to select adequate treatments, have attracted considerable attention in the clinical domain, especially from sepsis researchers. Existing sepsis DTR learning studies are mainly based on offline reinforcement learning (RL) approaches working on electronic healthcare records data. However, a trained policy may choose a treatment different from a human clinician's prescription. Furthermore, most of them do not consider: 1) heterogeneity in sepsis; 2) short-term transitions; and 3) the relationship between a patient's health state and the prescription. We propose a novel framework, an adaptive decision transformer for DTR (ADT2R), which recommends an optimal treatment action for each time step depending on the heterogeneity of the sepsis and a patient's evolving health states. Specifically, we devise a trajectory-optimization-based module to be trained with supervision for treatments and adaptively aggregate the multihead self-attentions by deliberating on various inherent time-varying patterns among sepsis patients. Furthermore, we estimate the patient's health state by adopting an actor-critic (AC) algorithm and inform the treatment recommendation by learning about its short-term changes. We validated the effectiveness of the proposed framework on the Medical Information Mart for Intensive Care III (MIMIC-III) dataset, an extensive intensive care database, by demonstrating performance comparable to the state-of-the-art methods.

Alzheimer's disease (AD) is a devastating neurodegenerative condition that precedes progressive and irreversible dementia; thus, predicting its progression over time is vital for clinical diagnosis and treatment. For this, numerous studies have implemented structural magnetic resonance imaging (MRI) to model AD progression, focusing on three integral aspects: 1) temporal variability; 2) incomplete observations; and 3) temporal geometric characteristics. However, many pioneer deep learning-based approaches addressing data variability and sparsity have yet to consider inherent geometrical properties sufficiently. These properties are integral to modeling as they correlate with brain region size, thickness, volume, and shape in AD progression. The ordinary differential equation-based geometric modeling method (ODE-RGRU) has recently emerged as a promising strategy for modeling time-series data by intertwining a recurrent neural network (RNN) and an ODE in Riemannian space. Despite its achievements, ODE-RGRU encounters limitations when extrapolating positive definite symmetric matrices from incomplete samples, leading to feature reverse occurrences that are particularly problematic, especially within the clinical facet. Therefore, this study proposes a novel geometric learning approach that models longitudinal MRI biomarkers and cognitive scores by combining three modules: topological space shift, ODE-RGRU, and trajectory estimation. We have also developed a training algorithm that integrates the manifold mapping with monotonicity constraints to reflect measurement transition irreversibility. We verify our proposed method's efficacy by predicting clinical labels and cognitive scores over time in regular and irregular settings. Furthermore, we thoroughly analyze our proposed framework through an ablation study.

Functional magnetic resonance imaging (fMRI) leverages the blood-oxygen-level-dependent (BOLD) signals to gauge functional brain activation. Specifically, task-fMRI has become a predominant tool to investigate specific cerebral regions associated with diverse cognitive processes. A myriad of studies employing task-fMRI have harnessed both static functional connectivity (FC) and dynamic functional connectivity (dFC) to identify task-related biomarkers. However, while FC and dFC have proven their efficacy, they mostly necessitate manual determination of factors, including window size, stride, and FC metrics, which potentially undermine their analysis reliability. In response to these challenges, one can detect and classify task-related patterns directly from the fMRI signals without using connectivity matrices (i.e., FC and dFC). Here, we propose a novel eigendecomposition-based attention mechanism (EAM) that emphasizes task-related regions and time points in the original signal space. By virtue of this approach, our proposed model allow us to effectively extract a refined integrated feature representation for identifying the brain cognitive states from the fMRI signals and also lessens the burden of making heuristic decisions in measuring both FC and dFC. We validate the superiority and effectiveness of our proposed method with comprehensive evaluations conducted on the Human Connectome Project (HCP) dataset, which covers seven distinct cognitive tasks.

Modeling non-euclidean data is drawing extensive attention along with the unprecedented successes of deep neural networks in diverse fields. Particularly, a symmetric positive definite matrix is being actively studied in computer vision, signal processing, and medical image analysis, due to its ability to learn beneficial statistical representations. However, owing to its rigid constraints, it remains challenging to optimization problems and inefficient computational costs, especially, when incorporating it with a deep learning framework. In this paper, we propose a framework to exploit a diffeomorphism mapping between Riemannian manifolds and a Cholesky space, by which it becomes feasible not only to efficiently solve optimization problems but also to greatly reduce computation costs. Further, for dynamic modeling of time-series data, we devise a continuous manifold learning method by systematically integrating a manifold ordinary differential equation and a gated recurrent neural network. It is worth noting that due to the nice parameterization of matrices in a Cholesky space, training our proposed network equipped with Riemannian geometric metrics is straightforward. We demonstrate through experiments over regular and irregular time-series datasets that our proposed model can be efficiently and reliably trained and outperforms existing manifold methods and state-of-the-art methods in various time-series tasks.

Medical image analysis (MedIA) has become an essential tool in medicine and healthcare, aiding in disease diagnosis, prognosis, and treatment planning, and recent successes in deep learning (DL) have made significant contributions to its advances. However, deploying DL models for MedIA in real-world situations remains challenging due to their failure to generalize across the distributional gap between training and testing samples—a problem known as domain shift. Researchers have dedicated their efforts to developing various DL methods to adapt and perform robustly on unknown and out-of-distribution (OOD) data distributions. This article comprehensively reviews domain generalization (DG) studies specifically tailored for MedIA. We provide a holistic view of how DG techniques interact within the broader MedIA system, going beyond methodologies to consider the operational implications on the entire MedIA workflow. Specifically, we categorize DG methods into data-level, feature-level, model-level, and analysis-level methods. We show how those methods can be used in various stages of the MedIA workflow with DL equipped from data acquisition to model prediction and analysis. Furthermore, we critically analyze the strengths and weaknesses of various methods, unveiling future research opportunities.

Recently, deep learning-based electroencephalogram (EEG) analysis and decoding have attracted widespread attention for monitoring the clinical condition of users and identifying their intention/emotion. Nevertheless, the existing methods generally model EEG signals with limited viewpoints or restricted concerns about the characteristics of the EEG signals, and thus represent complex spectro-/spatiotemporal patterns and suffer from high variability. In this work, we propose the novel EEG-oriented self-supervised learning methods and a novel deep architecture to learn rich representation, including information about the diverse spectral characteristics of neural oscillations, the spatial properties of electrode sensor distribution, and the temporal patterns of both the global and local viewpoints. Along with the proposed self-supervision strategies and deep architectures, we devise a feature normalization strategy to resolve the intra-/inter-subject variability problem. We demonstrate the validity of our proposed deep learning framework on the four publicly available datasets by conducting comparisons with the state of the art baselines. It is also noteworthy that we exploit the same network architecture for the four different EEG paradigms and outperform the comparison methods, thereby meeting the challenge of the task-dependent network architecture engineering in EEG-based applications.

Deep learning models based on resting-state functional magnetic resonance imaging (rs-fMRI) have been widely used to diagnose brain diseases, particularly autism spectrum disorder (ASD). Existing studies have leveraged the functional connectivity (FC) of rs-fMRI, achieving notable classification performance. However, they have significant limitations, including the lack of adequate information while using linear low-order FC as inputs to the model, not considering individual characteristics (i.e., different symptoms or varying stages of severity) among patients with ASD, and the non-explainability of the decision process. To cover these limitations, we propose a novel explainability-guided region of interest (ROI) selection (EAG-RS) framework that identifies non-linear high-order functional associations among brain regions by leveraging an explainable artificial intelligence technique and selects class-discriminative regions for brain disease identification. The proposed framework includes three steps: (i) inter-regional relation learning to estimate non-linear relations through random seed-based network masking, (ii) explainable connection-wise relevance score estimation to explore high-order relations between functional connections, and (iii) non-linear high-order FC-based diagnosis-informative ROI selection and classifier learning to identify ASD. We validated the effectiveness of our proposed method by conducting experiments using the Autism Brain Imaging Database Exchange (ABIDE) dataset, demonstrating that the proposed method outperforms other comparative methods in terms of various evaluation metrics. Furthermore, we qualitatively analyzed the selected ROIs and identified ASD subtypes linked to previous neuroscientific studies.

To understand the biological characteristics of neurological disorders with functional connectivity (FC), recent studies have widely utilized deep learning-based models to identify the disease and conducted post-hoc analyses via explainable models to discover disease-related biomarkers. Most existing frameworks consist of three stages, namely, feature selection, feature extraction for classification, and analysis, where each stage is implemented separately. However, if the results at each stage lack reliability, it can cause misdiagnosis and incorrect analysis in afterward stages. In this study, we propose a novel unified framework that systemically integrates diagnoses (i.e., feature selection and feature extraction) and explanations. Notably, we devised an adaptive attention network as a feature selection approach to identify individual-specific disease-related connections. We also propose a functional network relational encoder that summarizes the global topological properties of FC by learning the inter-network relations without pre-defined edges between functional networks. Last but not least, our framework provides a novel explanatory power for neuroscientific interpretation, also termed counter-condition analysis. We simulated the FC that reverses the diagnostic information (i.e., counter-condition FC): converting a normal brain to be abnormal and vice versa. We validated the effectiveness of our framework by using two large resting-state functional magnetic resonance imaging (fMRI) datasets, Autism Brain Imaging Data Exchange (ABIDE) and REST-meta-MDD, and demonstrated that our framework outperforms other competing methods for disease identification. Furthermore, we analyzed the disease-related neurological patterns based on counter-condition analysis.

Transfer learning has attracted considerable attention in medical image analysis because of the limited number of annotated 3-D medical datasets available for training data-driven deep learning models in the real world. We propose Medical Transformer, a novel transfer learning framework that effectively models 3-D volumetric images as a sequence of 2-D image slices. To improve the high-level representation in 3-D-form empowering spatial relations, we use a multiview approach that leverages information from three planes of the 3-D volume, while providing parameter-efficient training. For building a source model generally applicable to various tasks, we pretrain the model using self-supervised learning (SSL) for masked encoding vector prediction as a proxy task, using a large-scale normal, healthy brain magnetic resonance imaging (MRI) dataset. Our pretrained model is evaluated on three downstream tasks: 1) brain disease diagnosis; 2) brain age prediction; and 3) brain tumor segmentation, which are widely studied in brain MRI research. Experimental results demonstrate that our Medical Transformer outperforms the state-of-the-art (SOTA) transfer learning methods, efficiently reducing the number of parameters by up to approximately 92% for classification and regression tasks and 97% for segmentation task, and it also achieves good performance in scenarios where only partial training samples are used.

Large amounts of fMRI data are essential to building generalized predictive models for brain disease diagnosis. In order to conduct extensive data analysis, it is often necessary to gather data from multiple organizations. However, the site variation inherent in multisite resting-state functional magnetic resonance imaging (rs-fMRI) leads to unfavorable heterogeneity in data distribution, negatively impacting the identification of biomarkers and the diagnostic decision. Several existing methods have alleviated this shift of domain distribution (i.e., multisite problem). Statistical tuning schemes directly regress out site disparity factors from the data prior to model training. Such methods have a limitation in processing data each time through variance estimation according to the added site. In the model adjustment approaches, domain adaptation (DA) methods adjust the features or models of the source domain according to the target domain during model training. Thus, it is inevitable that it needs updating model parameters according to the samples of a target site, causing great limitations in practical applicability. Meanwhile, the approach of domain generalization (DG) aims to create a universal model that can be quickly adapted to multiple domains. In this study, we propose a novel framework for disease diagnosis that alleviates the multisite problem by adaptively calibrating site-specific features into site-invariant features. Specifically, it applies directly to samples from unseen sites without the need for fine-tuning. With a learning-to-learn strategy that learns how to calibrate the features under the various domain shift environments, our novel modulation mechanism extracts site-invariant features. In our experiments over the Autism Brain Imaging Data Exchange (ABIDE I and II) dataset, we validated the generalization ability of the proposed network by improving diagnostic accuracy in both seen and unseen multisite samples.

Face age transformation aims to convert reference images into synthesized images so that they portray the specified target ages. The crux of this task is to change only age-related areas of the given image while maintaining the age-irrelevant areas unchanged. Nevertheless, a common limitation among most existing models is the struggle to generate high-quality aging images that effectively consider both crucial properties. To address this problem, we propose a novel GAN-based face-aging framework that utilizes age-aware Guidance via Masking-Based Attention (GMBA). Specifically, we devise an age-aware guidance module to adjust age-relevant and age-irrelevant attributes within the image seamlessly. By virtue of its capability, it enables the model to produce realistic age-transformed images that certainly preserve the input's identities while delicately imposing age-related properties. Experimental results show that our proposed GMBA outperformed other state-of-the-art methods in terms of identity preservation and accurate age conversion, as well as providing superior visual quality for age-transformed images.

Existing studies on disease diagnostic models focus either on diagnostic model learning for performance improvement or on the visual explanation of a trained diagnostic model. We propose a novel learn-explain-reinforce (LEAR) framework that unifies diagnostic model learning, visual explanation generation (explanation unit), and trained diagnostic model reinforcement (reinforcement unit) guided by the visual explanation. For the visual explanation, we generate a counterfactual map that transforms an input sample to be identified as an intended target label. For example, a counterfactual map can localize hypothetical abnormalities within a normal brain image that may cause it to be diagnosed with Alzheimer's disease (AD). We believe that the generated counterfactual maps represent data-driven knowledge about a target task, i.e., AD diagnosis using structural MRI, which can be a vital source of information to reinforce the generalization of the trained diagnostic model. To this end, we devise an attention-based feature refinement module with the guidance of the counterfactual maps. The explanation and reinforcement units are reciprocal and can be operated iteratively. Our proposed approach was validated via qualitative and quantitative analysis on the ADNI dataset. Its comprehensibility and fidelity were demonstrated through ablation studies and comparisons with existing methods.

Sleep staging is essential for sleep assessment and plays a vital role as a health indicator. Many recent studies have devised various machine/deep learning methods for sleep staging. However, two key challenges hinder the practical use of those methods: 1) effectively capturing salient waveforms in sleep signals and 2) correctly classifying confusing stages in transitioning epochs. In this study, we propose a novel deep neural-network structure, TransSleep, that captures distinctive local temporal patterns and distinguishes confusing stages using two auxiliary tasks. In particular, TransSleep captures salient waveforms in sleep signals by an attention-based multiscale feature extractor and correctly classifies confusing stages in transitioning epochs, while modeling contextual relationships with two auxiliary tasks. Results show that TransSleep achieves promising performance in automatic sleep staging. The validity of TransSleep is demonstrated by its state-of-the-art performance on two publicly available datasets: 1) Sleep-EDF and 2) MASS. Furthermore, we performed ablations to analyze our results from different perspectives. Based on our overall results, we believe that TransSleep has immense potential to provide new insights into deep-learning-based sleep staging.

In recent years, deep learning-based feature representation methods have shown a promising impact on electroencephalography (EEG)-based brain-computer interface (BCI). Nonetheless, owing to high intra- and inter-subject variabilities, many studies on decoding EEG were designed in a subject-specific manner by using calibration samples, with no concern of its practical use, hampered by time-consuming steps and a large data requirement. To this end, recent studies adopted a transfer learning strategy, especially domain adaptation techniques. Among those, we have witnessed the potential of adversarial learning-based transfer learning in BCIs. In the meantime, it is known that adversarial learning-based domain adaptation methods are prone to negative transfer that disrupts learning generalized feature representations, applicable to diverse domains, for example, subjects or sessions in BCIs. In this article, we propose a novel framework that learns class-relevant and subject-invariant feature representations in an information-theoretic manner, without using adversarial learning. To be specific, we devise two operational components in a deep network that explicitly estimate mutual information between feature representations: 1) to decompose features in an intermediate layer into class-relevant and class-irrelevant ones and 2) to enrich class-discriminative feature representation. On two large EEG datasets, we validated the effectiveness of our proposed framework by comparing with several comparative methods in performance. Furthermore, we conducted rigorous analyses by performing an ablation study in regard to the components in our network, explaining our model's decision on input EEG signals via layer-wise relevance propagation, and visualizing the distribution of learned features via t-SNE.

Recently, deep learning-based electroencephalogram (EEG) analysis and decoding have gained widespread attention to monitor a user's clinical condition or identify his/her intention/emotion. Nevertheless, the existing methods mostly model EEG signals with limited viewpoints or restricted concerns about the characteristics of the EEG signals, thus suffering from representing complex spatio-spectro-temporal patterns as well as inter-subject variability. In this work, we propose novel EEG-oriented self-supervised learning methods to discover complex and diverse patterns of spatio-spectral characteristics and spatio-temporal dynamics. Combined with the proposed self-supervised representation learning, we also devise a feature normalization strategy to resolve an inter-subject variability problem via clustering. We demonstrated the validity of the proposed framework on three publicly available datasets by comparing with state-of-the-art methods.

Sleep staging is essential for sleep assessment and plays a vital role as one of the health indicators. It is challenging to correctly classify stage-transitioning epochs of sleep electroencephalography (EEG) because of their mixed signals of stages. To this end, recent studies exploited and devised various deep learning architectures. However, those are still suffering from confusing two or more stages, especially in stage-transitioning epochs. In this work, we propose a novel network architecture that takes advantage of two auxiliary classification tasks and exploits their outputs to adapt feature representations, thus effectively discriminating confusing stages. Specifically, one auxiliary task is an epoch-level stage classification to produce confidence scores about stages. The other is a stage-transition detection to learn inter-epoch relations. Using inferred information about stage-confusion at an epoch level and stage-transition across neighboring epochs helps learn more concrete representations for stage identification. We demonstrated and analyzed the validity of our proposed method over two publicly available datasets, achieving promising performances.

Alzheimer’s disease (AD) is widely aware as a neurodegenerative disease that is characterized as a leading cause of irreversible progressive dementia. From a clinical perspective, it is vital to forecast a patient’s progression over time. In that regard, there have been rigorous researches on AD progression modeling with structural magnetic resonance imaging (MRI). Methodologically, there are three major aspects of MRI modeling: (i) variability over time, (ii) sparseness in observations, and (iii) geometrical properties in temporal dynamics. While the existing deep-learning-based methods have addressed variability or sparsity in data, there is still a need to take into account the inherent geometrical properties. Recently, geometric modeling based on ordinary differential equations (ODE-RGRU) has shown its ability in various time-series data by combining an RNN and an ordinary differential equation (ODE) in symmetric positive definite (SPD) space. Despite the success of ODE-RGRU in time-series data modeling, it is limited to estimating the SPD matrix from sparse data with missing values. To this end, we propose a novel geometric learning framework for AD progression modeling to tackle the aforementioned issues simultaneously. And, we also propose training algorithms for manifold mapping on irregular and incomplete MRI and cognitive scores observations. Our proposed framework efficiently learns three major aspects of longitudinal MRI biomarker and cognitive scores by the manifold transformation module, ODE-RGRU, and missing value estimation module. We demonstrate the effectiveness of our method in experiments that forecast multi-class classification and cognitive scores over time. Additionally, we provide a multi-faceted analysis of the proposed method through an ablation study.

In this article, we formulate the problem of estimating and selecting task-relevant temporal signal segments from a single electroencephalogram (EEG) trial in the form of a Markov decision process and propose a novel reinforcement-learning mechanism that can be combined with the existing deep-learning-based brain–computer interface methods. To be specific, we devise an actor–critic network such that an agent can determine which timepoints need to be used (informative) or discarded (uninformative) in composing the intention-related features in a given trial, and thus enhancing the intention identification performance. To validate the effectiveness of our proposed method, we conduct experiments with a publicly available big motor imagery (MI) dataset and apply our novel mechanism to various recent deep-learning architectures designed for MI classification. Based on the exhaustive experiments, we observe that our proposed method helped achieve statistically significant improvements in performance.

Electronic health record (EHR) data are sparse and irregular as they are recorded at irregular time intervals, and different clinical variables are measured at each observation point. In this work, to handle irregular multivariate time-series data, we consider the human knowledge of the aspects to be measured and time to measure them in different situations, known as multi-view features, which are indirectly represented in the data. We propose a scheme to realize multi-view features integration learning via a self-attention mechanism. Specifically, we devise a novel multi-integration attention module (MIAM) to extract complex information that is inherent in irregular time-series data. We explicitly learn the relationships among the observed values, missing indicators, and time interval between the consecutive observations in a simultaneous manner. In addition, we build an attention-based decoder as a missing value imputer that helps empower the representation learning of the interrelations among multi-view observations for the prediction task this decoder operates only in the training phase so that the final model is implemented in an imputation-free manner. We validated the effectiveness of our method over the public MIMIC-III and PhysioNet challenge 2012 datasets by comparing with and outperforming the state-of-the-art methods in three downstream tasks i.e., prediction of the in-hospital mortality, prediction of the length of stay, and phenotyping. Moreover, we conduct a layer-wise relevance propagation (LRP) analysis based on case studies to highlight the explainability of the trained model.

Imaging genetics, one of the foremost emerging topics in the medical imaging field, analyzes the inherent relations between neuroimaging and genetic data. As deep learning has gained widespread acceptance in many applications, pioneering studies employed deep learning frameworks for imaging genetics. However, existing approaches suffer from some limitations. First, they often adopt a simple strategy for joint learning of phenotypic and genotypic features. Second, their findings have not been extended to biomedical applications, e.g., degenerative brain disease diagnosis and cognitive score prediction. Finally, existing studies perform insufficient and inappropriate analyses from the perspective of data science and neuroscience. In this work, we propose a novel deep learning framework to simultaneously tackle the aforementioned issues. Our proposed framework learns to effectively represent the neuroimaging and the genetic data jointly, and achieves state-of-the-art performance when used for Alzheimer's disease and mild cognitive impairment identification. Furthermore, unlike the existing methods, the framework enables learning the relation between imaging phenotypes and genotypes in a nonlinear way without any prior neuroscientific knowledge. To demonstrate the validity of our proposed framework, we conducted experiments on a publicly available dataset and analyzed the results from diverse perspectives. Based on our experimental results, we believe that the proposed framework has immense potential to provide new insights and perspectives in deep learning-based imaging genetics studies.

In this article, we focus on decomposing latent representations in generative adversarial networks or learned feature representations in deep autoencoders into semantically controllable factors in a semisupervised manner, without modifying the original trained models. Particularly, we propose factors' decomposer-entangler network (FDEN) that learns to decompose a latent representation into mutually independent factors. Given a latent representation, the proposed framework draws a set of interpretable factors, each aligned to independent factors of variations by minimizing their total correlation in an information-theoretic means. As a plug-in method, we have applied our proposed FDEN to the existing networks of adversarially learned inference and pioneer network and performed computer vision tasks of image-to-image translation in semantic ways, e.g., changing styles, while keeping the identity of a subject, and object classification in a few-shot learning scheme. We have also validated the effectiveness of the proposed method with various ablation studies in the qualitative, quantitative, and statistical examination.

Recently, deep learning, a branch of machine learning and data mining, has gained widespread acceptance in many applications thanks to its unprecedented successes. In this regard, pioneering studies employed deep learning frameworks for imaging genetics in virtue of their own representation caliber. But, existing approaches suffer from some limitations: (i) exploiting a simple concatenation strategy for joint analysis, (ii) a lack of extension to biomedical applications, and (iii) insufficient and inappropriate interpretations in the viewpoint of both data science and bio-neuroscience. In this work, we propose a novel deep learning framework to tackle the aforementioned issues simultaneously. Our proposed framework learns to effectively represent the neuroimaging and the genetic data jointly, and achieves state-of-the-art performance in its use for Alzheimer’s disease and mild cognitive impairment identification. Further, unlike the existing methods in the literature, the framework allows learning the relation between imaging phenotypes and genotypes in a nonlinear way without any prior neuroscientific knowledge. To demonstrate the validity of our proposed framework, we conducted experiments on a publicly available dataset and analyzed the results from diverse perspectives. Based on our experimental results, we believe that the proposed framework has a great potential to give new insights and perspectives in deep learning-based imaging genetics studies.

Electronic health records (EHRs) are characterized as nonstationary, heterogeneous, noisy, and sparse data; therefore, it is challenging to learn the regularities or patterns inherent within them. In particular, sparseness caused mostly by many missing values has attracted the attention of researchers who have attempted to find a better use of all available samples for determining the solution of a primary target task through defining a secondary imputation problem. Methodologically, existing methods, either deterministic or stochastic, have applied different assumptions to impute missing values. However, once the missing values are imputed, most existing methods do not consider the fidelity or confidence of the imputed values in the modeling of downstream tasks. Undoubtedly, an erroneous or improper imputation of missing variables can cause difficulties in the modeling as well as a degraded performance. In this study, we present a novel variational recurrent network that: 1) estimates the distribution of missing variables (e.g., the mean and variance) allowing to represent uncertainty in the imputed values; 2) updates hidden states by explicitly applying fidelity based on a variance of the imputed values during a recurrence (i.e., uncertainty propagation over time); and 3) predicts the possibility of in-hospital mortality. It is noteworthy that our model can conduct these procedures in a single stream and learn all network parameters jointly in an end-to-end manner. We validated the effectiveness of our method using the public data sets of MIMIC-III and PhysioNet challenge 2012 by comparing with and outperforming other state-of-the-art methods for mortality prediction considered in our experiments. In addition, we identified the behavior of the model that well represented the uncertainties for the imputed estimates, which showed a high correlation between the uncertainties and mean absolute error (MAE) scores for imputation.

As the population becomes older worldwide, accurate computer-aided diagnosis for Alzheimer's disease (AD) in the early stage has been regarded as a crucial step for neurodegeneration care in recent years. Since it extracts the low-level features from the neuroimaging data, previous methods regarded this computer-aided diagnosis as a classification problem that ignored latent featurewise relation. However, it is known that multiple brain regions in the human brain are anatomically and functionally interlinked according to the current neuroscience perspective. Thus, it is reasonable to assume that the extracted features from different brain regions are related to each other to some extent. Also, the complementary information between different neuroimaging modalities could benefit multimodal fusion. To this end, we consider leveraging the coupled interactions in the feature level and modality level for diagnosis in this paper. First, we propose capturing the feature-level coupled interaction using a coupled feature representation. Then, to model the modality-level coupled interaction, we present two novel methods: 1) the coupled boosting (CB) that models the correlation of pairwise coupled-diversity on both inconsistently and incorrectly classified samples between different modalities and 2) the coupled metric ensemble (CME) that learns an informative feature projection from different modalities by integrating the intrarelation and interrelation of training samples. We systematically evaluated our methods with the AD neuroimaging initiative data set. By comparison with the baseline learning-based methods and the state-of-the-art methods that are specially developed for AD/MCI (mild cognitive impairment) diagnosis, our methods achieved the best performance with accuracy of 95.0% and 80.7% (CB), 94.9% and 79.9% (CME) for AD/NC (normal control), and MCI/NC identification, respectively.

Electronic health records (EHR) have become an important source of a patient data but characterized by a variety of missing values. Using the variational inference of Bayesian framework, variational autoencoder (VAE), a deep generative model, has been shown to be efficient and accurate to capture the latent structure of complex high-dimensional data. Recently, it has been used for missing data imputation. In this paper, we propose a general framework that incorporates effective missing data imputation using VAE and multivariate time series prediction. We utilize the uncertainty obtained from the generative network of the VAE and employ uncertainty-aware attention in imputing the missing values. We evaluated the performance of our architecture on real-world clinical dataset (MIMIC-III) for in-hospital mortality prediction task. Our results showed higher performance than other competing methods in mortality prediction task.

In this paper, we propose a novel sparse regression method for Brain-Wide and Genome-Wide association study. Specifically, we impose a low-rank constraint on the weight coefficient matrix and then decompose it into two low-rank matrices, which find relationships in genetic features and in brain imaging features, respectively. We also introduce a sparse acyclic digraph with sparsity-inducing penalty to take further into account the correlations among the genetic variables, by which it can be possible to identify the representative SNPs that are highly associated with the brain imaging features. We optimize our objective function by jointly tackling low-rank regression and variable selection in a framework. In our method, the low-rank constraint allows us to conduct variable selection with the low-rank representations of the data; the learned low-sparsity weight coefficients allow discarding unimportant variables at the end. The experimental results on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset showed that the proposed method could select the important SNPs to more accurately estimate the brain imaging features than the state-of-the-art methods.

As the brain is a unique biological system that reflects the subtle distinctions in the mental attributes of individual humans, electroencephalographic (EEG) signals have been regarded as one of the most promising and potent biometric signals for discriminating between individuals. However, existing EEG-based user-recognition methods present only a limited range of individual distinctions. In this paper, we propose a novel system of decoding cognitive EEG signals for individual identification with high accuracy. Specifically, we investigate the feasibility of our system, which can recognize an individual based on the discriminative patterns of source-level causal connectivity among brain regions, estimated from scalp-level EEG signals. The EEG signals were produced by a steady-state visual evoked potential-inducing grid-shaped top-down paradigm. This system can, in principle, use top-down cognitive features analyzed by individuals' differently characterized neurodynamic causal connectivities. In this paper, we achieved a maximal accuracy of 98.60% on average in 20 subjects, for whom we estimated causal connectivity in 16 brain regions using 5-s intervals of EEG signals. Our system shows promising initial results toward building a practical identification technology able to recognize individuals by means of brain neurodynamics.

Alzheimer's disease (AD) is a progressive and irreversible neurodegenerative disorder that has recently seen serious increase in the number of affected subjects. In the last decade, neuroimaging has been shown to be a useful tool to understand AD and its prodromal stage, amnestic mild cognitive impairment (MCI). The majority of AD/MCI studies have focused on disease diagnosis, by formulating the problem as classification with a binary outcome of AD/MCI or healthy controls. There have recently emerged studies that associate image scans with continuous clinical scores that are expected to contain richer information than a binary outcome. However, very few studies aim at modeling multiple clinical scores simultaneously, even though it is commonly conceived that multivariate outcomes provide correlated and complementary information about the disease pathology. In this article, we propose a sparse multi-response tensor regression method to model multiple outcomes jointly as well as to model multiple voxels of an image jointly. The proposed method is particularly useful to both infer clinical scores and thus disease diagnosis, and to identify brain subregions that are highly relevant to the disease outcomes. We conducted experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, and showed that the proposed method enhances the performance and clearly outperforms the competing solutions.

In this paper, we propose a novel method for unusual human activity detection in crowded scenes. Specifically, rather than detecting or segmenting humans, we devised an efficient method, called a motion influence map, for representing human activities. The key feature of the proposed motion influence map is that it effectively reflects the motion characteristics of the movement speed, movement direction, and size of the objects or subjects and their interactions within a frame sequence. Using the proposed motion influence map, we further developed a general framework in which we can detect both global and local unusual activities. Furthermore, thanks to the representational power of the proposed motion influence map, we can localize unusual activities in a simple manner. In our experiments on three public datasets, we compared the performances of the proposed method with that of other state-of-the-art methods and showed that the proposed method outperforms these competing methods.

In this paper; we propose a novel crowd behavior representation method to detect abnormal behaviors in videos. An adaptive optical flow filtering method is proposed to utilize low-level optical flow informations. Furthermore, a simple framework is developed to detect and to localize abnormal crowd behavior using adaptive optical flow filtering result. The proposed method is more robust than other modeling methods in representing different behaviors. In this model, a normal behavior is presented by the general value. Some outliers in the temporal domain or spatial domain are presented by a higher value. Spatio-temporal cuboids are extracted from the filtering result to present the likelihood of anomaly in the frame. Experimental evaluations are performed on two public datasets with comparison to the provisos abnormal behavior detection methods in the literature. Experimental results show that the proposed methods outperform previous abnormal behavior detection techniques in the literature.

Recently, there has been a great interest in computer-aided Alzheimer's Disease (AD) and Mild Cognitive Impairment (MCI) diagnosis. Previous learning based methods defined the diagnosis process as a classification task and directly used the low-level features extracted from neuroimaging data without considering relations among them. However, from a neuroscience point of view, it's well known that a human brain is a complex system that multiple brain regions are anatomically connected and functionally interact with each other. Therefore, it is natural to hypothesize that the low-level features extracted from neuroimaging data are related to each other in some ways. To this end, in this paper, we first devise a coupled feature representation by utilizing intra-coupled and inter-coupled interaction relationship. Regarding multi-modal data fusion, we propose a novel coupled boosting algorithm that analyzes the pairwise coupled-diversity correlation between modalities. Specifically, we formulate a new weight updating function, which considers both incorrectly and inconsistently classified samples. In our experiments on the ADNI dataset, the proposed method presented the best performance with accuracies of 94.7% and 80.1% for AD vs. Normal Control (NC) and MCI vs. NC classifications, respectively, outperforming the competing methods and the state-of-the-art methods.

Recent studies on Alzheimer's Disease (AD) or its prodromal stage, Mild Cognitive Impairment (MCI), diagnosis presented that the tasks of identifying brain disease status and predicting clinical scores based on neuroimaging features were highly related to each other. However, these tasks were often conducted independently in the previous studies. Regarding the feature selection, to our best knowledge, most of the previous work considered a loss function defined as an element-wise difference between the target values and the predicted ones. In this paper, we consider the problems of joint regression and classification for AD/MCI diagnosis and propose a novel matrix-similarity based loss function that uses high-level information inherent in the target response matrix and imposes the information to be preserved in the predicted response matrix. The newly devised loss function is combined with a group lasso method for joint feature selection across tasks, i.e., clinical scores prediction and disease status identification. We conducted experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, and showed that the newly devised loss function was effective to enhance the performances of both clinical score prediction and disease status identification, outperforming the state-of-the-art methods.

The problem of viewpoint changes is an important issue in the study of human action recognition. In this paper, we propose the use of spatial features in a spatiotemporal self-similarity matrix (SSM) based on action recognition that is robust in viewpoint changes from depth sequences. The spatial features represent a discriminative density of 3D point clouds in a 3D grid. We construct the spatiotemporal SSM for the spatial features that change along with frames. To obtain the spatiotemporal SSM, we compute the Euclidean distance of each spatial feature between two frames. The spatiotemporal SSM represents similarity of human action robust in viewpoint changes. Our proposed method is robust in viewpoint changes and various length of action sequence. This method is evaluated on ACTA2 dataset containing the multi-view RGBD human action data, and MSRAction3D dataset. In the experimental validation, the spatiotemporal SSM is a good solution for the problem of viewpoint changes in a depth sequence.

As there has been a paradigm shift in the learning load from a human subject to a computer, machine learning has been considered as a useful tool for Brain-Computer Interfaces (BCIs). In this paper, we propose a novel Bayesian framework for discriminative feature extraction for motor imagery classification in an EEG-based BCI in which the class-discriminative frequency bands and the corresponding spatial filters are optimized by means of the probabilistic and information-theoretic approaches. In our framework, the problem of simultaneous spatiospectral filter optimization is formulated as the estimation of an unknown posterior probability density function (pdf) that represents the probability that a single-trial EEG of predefined mental tasks can be discriminated in a state. In order to estimate the posterior pdf, we propose a particle-based approximation method by extending a factored-sampling technique with a diffusion process. An information-theoretic observation model is also devised to measure discriminative power of features between classes. From the viewpoint of classifier design, the proposed method naturally allows us to construct a spectrally weighted label decision rule by linearly combining the outputs from multiple classifiers. We demonstrate the feasibility and effectiveness of the proposed method by analyzing the results and its success on three public databases.

In this paper, we propose a novel method of online learning of sparse pseudo-data, representative of the whole training data, for Gaussian Process (GP) regressions. We call the proposed method Incremental Sparse Pseudo-input Gaussian Process (ISPGP) regression. The proposed ISPGP algorithm allows for training from either a huge amount of training data by scanning through it only once or an online incremental training dataset. Thanks to the nature of the incremental learning algorithm, the proposed ISPGP algorithm can theoretically work with infinite data to which the conventional GP or SPGP algorithm is not applicable. From our experimental results on the KIN40K dataset, we can see that the proposed ISPGP algorithm is comparable to the conventional GP algorithm using the same number of training data. Although the proposed ISPGP algorithm performs slightly worse than Snelson and Ghahramani's SPGP algorithm, the level of performance degradation is acceptable.

In this paper, we devise a novel method that incrementally learns pseudo-data, which represent the whole training data set for Gaussian Process (GP) regression. The method involves sparse approximation of the GP by extending the work of Snelson and Ghahramani. We call the proposed method Incremental Sparse Pseudo-input Gaussian Process (ISPGP) regression. Unlike the Snelson and Ghahramani's work, the proposed ISPGP algorithm allows for training from either a huge amount of training data by scanning through it only once or an online incremental training data set. We also design a likelihood weighting scheme to incrementally determine pseudo-data while maintaining the representational power. Due to the nature of the incremental learning algorithm, the proposed ISPGP algorithm can theoretically work with infinite data to which the conventional GP or Sparse Pseudo-input Gaussian Process (SPGP) algorithm is not applicable. From our experimental results on the KIN40K data set, we can see that the proposed ISPGP algorithm is comparable to the conventional GP algorithm using the same number of training data. It also significantly reduces the computational cost and memory requirement in regression and is scalable to a large training data set without significant performance degradation. Although the proposed ISPGP algorithm performs slightly worse than Snelson and Ghahramani's SPGP algorithm, the level of performance degradation is acceptable.

EEG-based motor imagery classification has been widely studied for Brain-Computer Interfaces (BCIs) due to its asynchronous and continuous elicitation and its great potential to many applications. Many research groups have devoted their efforts to either the frequency band selection or optimal spatial filters learning via the Common Spatial Pattern (CSP) algorithm. However, since the spectral filtering and the spatial filtering are generally operated in order in a motor imagery classification system the optimization of the spatial filters and the spectral filters should be considered simultaneously in a unified framework. In this paper, we propose a novel probabilistic approach for the spatio-spectral filters optimization in an EEG-based BCI with a particle-filter algorithm and mutual information between feature vectors and class labels. There are two main contributions of the proposed method. The one is that it finds the optimal frequency bands that maximally discriminate the feature vectors of two classes in terms of an information theoretic approach. The other is that we construct a spectrally-weighted label decision rule by linearly combining the outputs from multiple SVMs, one for each frequency band, with the weight of the corresponding frequency band. From our experiments with two publicly available dataset, we confirm that the proposed method outperforms the other competing methods.

We propose a novel method of analyzing human interactions based on the walking trajectories of human subjects, which provide elementary and necessary components for understanding and interpretation of complex human interactions in visual surveillance tasks. Our principal assumption is that an interaction episode is composed of meaningful small unit interactions, which we call “sub-interactions”. We model each sub-interaction by a dynamic probabilistic model and propose a modified factorial hidden Markov model (HMM) with factored observations. The complete interaction is represented with a network of dynamic probabilistic models (DPMs) by an ordered concatenation of sub-interaction models. The rationale for this approach is that it is more effective in utilizing common components, i.e., sub-interaction models, to describe complex interaction patterns. By assembling these sub-interaction models in a network, possibly with a mixture of different types of DPMs, such as standard HMMs, variants of HMMs, dynamic Bayesian networks, and so on, we can design a robust model for the analysis of human interactions. We show the feasibility and effectiveness of the proposed method by analyzing the structure of network of DPMs and its success on four different databases: a self-collected dataset, Tsinghua University's dataset, the public domain CAVIAR dataset, and the Edinburgh Informatics Forum Pedestrian dataset.

In this paper, we propose a novel method for analyzing human interactions based on the walking trajectories of human subjects. Our principal assumption is that an interaction episode is composed of meaningful smaller unit interactions, which we call `sub-interactions.' The whole interaction is represented by an ordered concatenation or a network of sub-interaction models. From the experiments, we could confirm the effectiveness and robustness of the proposed method by analyzing the internal work of an interaction network and comparing the performance with other previous approaches.

In this paper, we describe a dynamic Bayesian network or DBN based approach to both two-hand gestures and one-hand gestures. Unlike wired glove-based approaches, the success of camera-based methods depends greatly on image processing and feature extraction results. So the proposed method of DBN-based inference is preceded by fail-safe steps of motion tracking. Then a new gesture recognition model for a set of both one-hand and two-hand gestures is proposed based on the dynamic Bayesian network framework which makes it easy to represent the relationship among features and incorporate new information to the model. In an experiment with ten isolated gestures, we obtained a recognition rate upwards of 99.59% with cross validation. The proposed model is believed to have a strong potential for successful applications to other related problems such as sign languages.

In this paper, we propose a new gesture recognition model for a set of both one-hand and two-hand gestures based on the dynamic Bayesian network framework which makes it easy to represent the relationship among features and incorporate new information to the model. Unlike the coupled HMM, the proposed model has room for common hidden variables which are believed to be shared between two variables. In an experiment with ten isolated gestures, we obtained a recognition rate upwards of 99.59% with leave-one-out cross validation. The proposed model is believed to have a strong potential for successful applications to other related problems such as sign languages.