Despite the remarkable progress in the development of predictive models for healthcare, applying these algorithms on a large scale has been challenging. Algorithms trained on a particular task, based on specific data formats available in a set of medical records, tend to not generalize well to other tasks or databases in which the data fields may differ. To address this challenge, we propose General Healthcare Predictive Framework (GenHPF), which is applicable to any EHR with minimal preprocessing for multiple prediction tasks. GenHPF resolves heterogeneity in medical codes and schemas by converting EHRs into a hierarchical textual representation while incorporating as many features as possible. To evaluate the efficacy of GenHPF, we conduct multi-task learning experiments with single-source and multi-source settings, on three publicly available EHR datasets with different schemas for 12 clinically meaningful prediction tasks. Our framework significantly outperforms baseline models that utilize domain knowledge in multi-source learning, improving average AUROC by 1.2%P in pooled learning and 2.6%P in transfer learning while also showing comparable results when trained on a single EHR dataset. Furthermore, we demonstrate that self-supervised pretraining using multi-source datasets is effective when combined with GenHPF, resulting in a 0.6%P AUROC improvement compared to models without pretraining. By eliminating the need for preprocessing and feature engineering, we believe that this work offers a solid framework for multi-task and multi-source learning that can be leveraged to speed up the scaling and usage of predictive algorithms in healthcare.

Electroencephalography (EEG) is a useful way to implicitly monitor the user's perceptual state during multimedia consumption. One of the primary challenges for the practical use of EEG-based monitoring is to achieve a satisfactory level of accuracy in EEG classification. Connectivity between different brain regions is an important property for the classification of EEG. However, how to define the connectivity structure for a given task is still an open problem, because there is no ground truth about how the connectivity structure should be in order to maximize the classification performance. In this paper, we propose an end-to-end neural network model for EEG-based emotional video classification, which can extract an appropriate multi-layer graph structure and signal features directly from a set of raw EEG signals and perform classification using them. Experimental results demonstrate that our method yields improved performance in comparison to the existing approaches where manually defined connectivity structures and signal features are used. Furthermore, we show that the graph structure extraction process is reliable in terms of consistency, and the learned graph structures make much sense in the viewpoint of emotional perception occurring in the brain.

Functional near-infrared spectroscopy (fNIRS) is a brain imaging method introduced relatively recently, which is promising to implement brain-computer interfaces. However, there is still a lack of research on fNIRS signal classification, particularly that focusing on improved machine learning techniques for non-motor tasks. In this paper, we propose a novel deep learning method using brain connectivity for resting-state fNIRS signal classification. Our method is based on the powerful modeling capability of the convolutional neural network that learns the brain connectivity patterns residing in the fNIRS signal. In particular, we present a new data augmentation method that can overcome the scarcity of fNIRS data. Experimental results of subject-independent classification of flourishing levels demonstrate the superiority of our approach to conventional approaches. It is also shown that the data augmentation strategy is effective for improving classification performance.

In this paper, we deal with the issue of implicit monitoring of perceptual responses to product design through electroencephalography (EEG) and eye tracking. Four evaluation factors, namely, preference, luxury, complexity, and harmony are considered to investigate how people perceive the car design. In particular, the quantified perceptual responses are predicted based on EEG and gaze data. Average root-mean-square errors of 0.210 and 1.215 are obtained from subject-dependent and subject-independent regressions on a 7-point score scale, respectively, which demonstrates that perception of car design can be predicted via implicit monitoring.

This paper proposes a novel deep learning method for extraction of the conjunctive information that describes the relationship between signals in multi-sensor systems to enhance the performance of the given classification task. The signals obtained from different sensors included in the multi-sensor systems are closely related. Handcrafted metrics have been used to extract the relationship between the signals in some work, which is hardly optimal for the given task. Our proposed method learns the pair-wise relationship from data to maximize the performance of the given task, which is fully data-driven, multi-aspect, and target-oriented. We demonstrate the effectiveness of the proposed method on a toy example and two real-world problems, i.e., activity recognition using accelerometer signals and emotional video classification using brain signals.

This paper proposes a novel graph signal-based deep learning method for electroencephalography (EEG) and its application to EEG-based video identification. We present new methods to effectively represent EEG data as signals on graphs, and learn them using graph convolutional neural networks. Experimental results for video identification using EEG responses obtained while watching videos show the effectiveness of the proposed approach in comparison to existing methods. Effective schemes for graph signal representation of EEG are also discussed.

Emotion recognition based on electroencephalography (EEG) has received attention as a way to implement human-centric services. However, there is still much room for improvement, particularly in terms of the recognition accuracy. In this paper, we propose a novel deep learning approach using convolutional neural networks (CNNs) for EEG-based emotion recognition. In particular, we employ brain connectivity features that have not been used with deep learning models in previous studies, which can account for synchronous activations of different brain regions. In addition, we develop a method to effectively capture asymmetric brain activity patterns that are important for emotion recognition. Experimental results confirm the effectiveness of our approach.

Evaluation of quality of experience (Qo E) based on electroencephalography (EEG) has received great attention due to its capability of real-time Qo E monitoring of users. However, it still suffers from rather low recognition accuracy. In this paper, we propose a novel method using deep neural networks toward improved modeling of EEG and thereby improved recognition accuracy. In particular, we aim to model spatio-temporal characteristics relevant for QoE analysis within learning models. The results demonstrate the effectiveness of the proposed method.

Electroencephalography (EEG) has attracted much attention because it allows to monitor user states in real time and is applicable to applications for improvement of multimedia user experience. The repeatability of EEG-based perceptual response analysis is critical for the reliability of such applications, which has not been sufficiently addressed in previous studies. In this paper, we evaluate the repeatability of EEG-based image quality assessment. We repeatedly perform the same experiment for three successive days. Then, we design a classification system that discriminates the quality of images based on EEG, whose performance is examined across the data collected in different days. We reveal that the repeatability of image quality evaluation using EEG decreases as time goes and the memory effect intensifies the loss of repeatability.

The exponential growth of popularity of multimedia has led to needs for user-centric adaptive applications that manage multimedia content more effectively. Implicit analysis, which examines users' perceptual experience of multimedia by monitoring physiological or behavioral cues, has potential to satisfy such demands. Particularly, physiological signals categorized into cerebral physiological signals (electroencephalography, functional magnetic resonance imaging, and functional near-infrared spectro-scopy) and peripheral physiological signals (heart rate, respiration, skin temperature, etc.) have recently received attention along with notable development of wearable physiological sensors. In this paper, we review existing studies on physiological signal analysis exploring perceptual experience of multimedia. Furthermore, we discuss current trends and challenges.

This paper explores the relations between perspectives of navigation and visual perception in 3D virtual space, by analyzing avatar-based navigation with the eye-gaze data. We examine how different perspectives and types of avatars affect the users' scopes of visual perception within 3D virtual environments. Throughout this research, we attempt to draw possible connections between the perspectives and cognitive patterns of visual perception. We propose that manipulating perspectives of avatars or those of users has immediate effects on the users' scopes of visual perception and patterns of visual attention.

In this paper we explore the relations between perspectives of navigation and electroencephalogram (EEG) in 3D virtual space. We analyze three types of navigation with EEG recordings and examine how the perspectives affect the users' electrical activities in their brains. Via a small-scale experiment, we find that the influence of peripersonal space is altered by the perspective, and it can be observed via EEG monitoring. These results have interesting implications on virtual reality applications where a sense of agency, or a peripersonal task takes important roles.

High dynamic range (HDR) imaging has attracted attention as a new technology for immersive multimedia experience. In comparison to conventional low dynamic range (LDR) contents, HDR contents are expected to provide better quality of experience (QoE). In this paper, we investigate implicit QoE measurement of tone-mapped HDR videos by using connectivity-based EEG features that convey information about simultaneous activations of different brain regions and thus can explain better the cognitive process than the conventional features using single channel powers. Through the experiment classifying EEG signals into tone-mapped HDR and LDR, it is shown that the connectivity features, particularly those representing directed information flows between brain regions, are effective in both subject-dependent and subject-independent scenarios.