Virtual reality (VR) has become an increasingly popular way for learning and training. The assessment of the amount of mental effort, or cognitive load required to perform a task, is essential to create adaptive VR experiences. In this work, we conducted a large-scale study (N=738) to collect behavioral and physiological measures under different cognitive load conditions in a VR environment, and developed a novel machine learning solution to predict cognitive load in real time. Our model predicts cognitive load as a continuous value in the range from 0 to 1, where 0 and 1 correspond to the lowest and highest reported cognitive loads across all participants. On top of the point estimation of cognitive load, our model quantifies prediction uncertainty using a prediction interval. We propose a novel dual-branch attention model to accurately predict the cognitive load. We achieve a MAE (Mean Absolute Error) of 0.11. The result indicates that, with a combination of behavioral and physiological indicators, we can reliably predict cognitive load in real-time, without calibration. To support further research, we are releasing a test dataset comprising data from 100 participants for use by researchers and developers interested in machine learning, virtual reality, learning & memory, cognition, or psychophysiology. This dataset includes recordings from multiple sensors (including pupillometry, eye-tracking, and pulse plethysmography), self-reported cognitive effort, behavioral task performance, and demographic information on the sample.

Electrocardiogram (ECG) is one of the critical diagnostic tools in healthcare. Various deep learning models, except Transformers, have been explored and applied to map ECG patterns to heart abnormalities. Transformer models have been adopted from natural language processing to computer vision with advanced features. Most recently, vision transformers show exceptional performances, even on moderate-scale datasets. However, naively applying vision transformers on electrocardiogram datasets leads to poor results. In this paper, we propose a novel network called Beat-aligned Transformer (BaT), a hierarchical Transformer that sufficiently exploits the cyclicity of ECG. We organize and treat an input ECG as multiple aligned beats instead of a single time series. In the BaT, shifted-window-based Transformer blocks (SW Block) are adopted to learn the representation for each beat, and aggregation blocks are designed to exchange information among the beat representations. Nested SW Blocks and aggregation blocks form a beat-aware hierarchical structure of BaT. In this way, the new data format and the BaT hierarchical structure boost Transformer performance on ECG classification. From the experiments on public ECG datasets, we observe BaT outperforms other Transformer-based models and achieves competitive performance compared with other state-of-the-art methods.

Deep neural networks (DNNs) provide the best of class solutions to many supervised tasks due to their powerful function fitting capabilities. However, it is challenging to handle data bias, such as label noise and class imbalance, when applying DNNs to solve real-world problems. Sample reweighting is a popular strategy to tackle data bias, which assigns higher weights to informative samples or samples with clean labels. However, conventional reweighting methods require prior knowledge of the distribution information of data bias, which is intractable in practice. In recent years, meta-learning-based methods have been proposed to learn to assign weights to training samples adaptively by using their online training loss or gradient directions. However, the latent bias distribution cannot be adequately characterized in an online fashion. The online loss distribution changes over the training procedure, making it even harder to perform the sample weight learning. In contrast to past methods, we propose a two-stage training strategy to tackle the above problems. In the first stage, the loss sequences of samples are collected. In the second stage, a subnet with convolutional layers is utilized to learn the mapping from offline sample loss sequence to sample weight adaptively. Guided by a small unbiased meta dataset, this subnet is optimized iteratively with the main classifier network in a meta-learning manner. Empirical results show that our method, called Meta Reweighting with Offline Loss Sequence (MROLS), outperforms state-of-the-art reweighting techniques on most benchmarks. Moreover, the weights of training samples learned via MROLS can be well utilized by other classifiers, which can directly enhance the standard training schema. Our source code is available at https://github.com/Neronjust2017/MROLS.

Hand-crafted features have been proven useful in solving the electrocardiograph~(ECG) classification problem. The features rely on domain knowledge and carry clinical meanings. However, the construction of the features requires tedious fine tuning in practice. Lately, a set of end-to-end deep neural network models have been proposed and show promising results in ECG classification. Though effective, such models learn patterns which usually mismatch human's concept, and thereby it is hard to get a convincing explanation with interpretation methods. This limitation significantly narrows the applicability of deep models, considering it is difficult for cardiologists to accept the unexplainable results from deep learning. To alleviate such limitation, we are bringing the best from the two worlds and propose a domain knowledge guided deep neural network. Specifically, we utilize a deep residual network as a classification framework, within which key feature ~(P-wave and R-peak position) reconstruction tasks are adopted to incorporate domain knowledge in the learning process. The reconstruction tasks make the model pay more attention to key feature points within ECG. Furthermore, we utilize occlusion method to get visual interpretation and design a visualization at both heartbeat level and feature point level. Our experiments show the superior performance of the proposed ECG classification methods compared to the model without P-wave and R-peak tasks, and the patterns learnt by our model is more explainable.

Electroencephalogram(EEG) is a test that detect brain activities using multiple electrodes placed on the scalp. Multiple channels of EEG signals are recorded through the electrodes and are widely used in applications such as neurological disease diagnosis, emotion recognition, and behavior modeling. Recently, deep learning methods have been applied to classify EEG signals, where the different EEG channels are almost treated as a 2D grid input to the machine learning model. This data formation doesn't consider The complex connection among the EEG channels is not considered in such data formation. In our work, we treat EEG signals as frames of graph, and propose an end-to-end edge-aware spatio-temporal graph convolutional neural network for EEG classification. Specifically, we iteratively apply graph convolutional layer spatially and standard convolutional layer temporally. Since there is no prior knowledge about the exact connection among EEG channels, in our model, we initialize the connection as complete graph and apply learnable mask to capture graph structure at different levels. Furthermore, we also propose an iterative method based on information aggregation in graph convolution mechanism to reveal the latent graph structure. Empirical evaluation shows that our model achieves superior performance over state-of-the-art methods for EEG classification, and the learnt and revealed latent EEG graph structure is verified to be meaningful by neuroscientists.

Medical images are widely used in the medical domain for the diagnosis and treatment of diseases. Reading a medical image and summarizing its insights is a routine, yet nonetheless time-consuming task, which often represents a bottleneck in the clinical diagnosis process. Automatic report generation can relieve the issues. However, generating medical reports presents two major challenges: (i) it is hard to accurately detect all the abnormalities simultaneously, especially the rare diseases; (ii) a medical image report consists of many paragraphs and sentences, which are longer than natural image captions. We present a new framework to accurately detect the abnormalities and automatically generate medical reports. The report generation model is based on hierarchical recurrent neural network (HRNN). We introduce a topic matching mechanism to HRNN, so as to make generated reports more accurate and diverse. The soft attention mechanism is also introduced to HRNN model. Experimental results on two image-paragraph pair datasets show that our framework outperforms all the state-of-art methods.

Lately, thanks to the huge amount of Electronic Health Records (EHR) data, deep learning models have been successfully applied to a variety of clinical prediction problems. Existing state-of-the-art clinical predicting models are usually built with recurrent neural network (RNN) and attention mechanism. However, such RNN based approaches mainly suffer from three limitations on clinical predictions, which if addressed would significantly widen their applicability. (i) Accuracy: The performance of RNN based models drops fast when the length of EHR sequences increases. (ii) Interpretability: The prediction results of RNN based models are hard to interpret due to the nature of deep models. (iii) Efficiency: The sequential property of RNN based models makes the parallelization of computation impossible, and accordingly hurts the efficiency of such models in practice. In this paper, we propose an efficient attention-based model to address the above three challenges simultaneously. In the context of heart failure prediction task, we demonstrate the interpretation capability of our model by visualizing relative connections between events and prediction result, and the high computational efficiency comparing to other baseline methods. Meanwhile, we show that the accuracy of our prediction model is comparable or better than those of other state-of-the-art prediction models in healthcare applications.

The need for short-text classification arises in many text mining applications particularly health care applications. In such applications shorter texts mean linguistic ambiguity limits the semantic expression, which in turns would make typical methods fail to capture the exact semantics of the scarce words. This is particularly true in health care domains when the text contains domain-specific or infrequently appearing words, whose embedding can not be easily learned due to the lack of training data. Deep neural network has shown great potentials in boost the performance of such problems according to its strength on representation capacity. In this paper, we propose a bidirectional long short-term memory (BI-LSTM) recurrent network to address the short-text classification problem that can be used in two settings. Firstly when a knowledge dictionary is available we adopt the well-known attention mechanism to guide the training of network using the domain knowledge in the dictionary. Secondly, to address the cases when domain knowledge dictionary is not available, we present a multi-task model to jointly learn the domain knowledge dictionary and do the text classification task simultaneously. We apply our method to a real-world interactive healthcare system and an extensively public available ATIS dataset. The results show that our model can positively grasp the key point of the text and significantly outperforms many state-of-the-art baselines.

Active learning aims to reduce manual labeling efforts by proactively selecting the most informative unlabeled instances to query. In real-world scenarios, it's often more practical to query a batch of instances rather than a single one at each iteration. To achieve this we need to keep not only the informativeness of the instances but also their diversity. Many heuristic methods have been proposed to tackle batch mode active learning problems, however, they suffer from two limitations which if addressed would significantly improve the query strategy. Firstly, the similarity amongst instances is simply calculated using the feature vectors rather than being jointly learned with the classification model. This weakens the accuracy of the diversity measurement. Secondly, these methods usually exploit the decision boundary by querying the data points close to it. However, this can be inefficient when the labeled set is too small to reveal the true boundary. In this paper, we address both limitations by proposing a deep neural network based algorithm. In the training phase, a pairwise deep network is not only trained to perform classification, but also to project data points into another space, where the similarity can be more precisely measured. In the query selection phase, the learner selects a set of instances that are maximally uncertain and minimally redundant (exploitation), as well as are most diverse from the labeled instances (exploration). We evaluate the effectiveness of the proposed method on a variety of classification tasks: MNIST classification, opinion polarity detection, and heart failure prediction. Our method outperforms the baselines with both higher classification accuracy and faster convergence rate.

Comparing and identifying similar patients is a fundamental task in medical domains - an efficient technique can, for example, help doctors to track patient cohorts, compare the effectiveness of treatments, or predict medical outcomes. The goal of patient similarity learning is to derive a clinically meaningful measure to evaluate the similarity amongst patients represented by their key clinical indicators. However, it is challenging to learn such similarity, as medical data are usually high dimensional, heterogeneous, and complex. In addition, a desirable patient similarity is dependent on particular clinical settings, which implies supervised learning scheme is more useful in medical domains. To address these, in this paper we present a novel similarity learning approach formulated as the generalized Mahalanobis similarity function with pairwise constraints. Considering there always exists some features non-discriminative and contains redundant information, we encode a low-rank structure to our similarity function to perform feature selection. We evaluate the proposed model on both UCI benchmarks and a real clinical dataset for several medical tasks, including patient retrieval, classification, and cohort discovery. The results show that our similarity model significantly outperforms many state-of-the-art baselines, and is effective at removing noisy or redundant features.

Evaluating the clinical similarities between pairwisepatients is a fundamental problem in healthcare informatics. Aproper patient similarity measure enables various downstreamapplications, such as cohort study and treatment comparative effectiveness research. One major carrier for conductingpatient similarity research is the Electronic Health Records(EHRs), which are usually heterogeneous, longitudinal, andsparse. Though existing studies on learning patient similarityfrom EHRs have shown being useful in solving real clinicalproblems, their applicability is limited due to the lack of medicalinterpretations. Moreover, most previous methods assume avector based representation for patients, which typically requiresaggregation of medical events over a certain time period. As aconsequence, the temporal information will be lost. In this paper, we propose a patient similarity evaluation framework based ontemporal matching of longitudinal patient EHRs. Two efficientmethods are presented, unsupervised and supervised, both ofwhich preserve the temporal properties in EHRs. The supervisedscheme takes a convolutional neural network architecture, andlearns an optimal representation of patient clinical recordswith medical concept embedding. The empirical results on real-world clinical data demonstrate substantial improvement overthe baselines.

Current simulations of turbulent flames are instrumented with particles to capture the dynamic behavior of combustion in next-generation engines. Categorizing the set of many millions of particles, each of which is featured with a history of its movement positions and changing thermo-chemical states, helps understand the turbulence mechanism. We introduce a dual-space method to analyze such data, starting by clustering the time series curves in the phase space of the data, and then visualizing the corresponding trajectories of each cluster in the physical space. To cluster time series curves, we adopt a model-based clustering technique in a two-stage scheme. In the first stage, the characteristics of shape and relative position are particularly concerned in classifying the time series curves, and in the second stage, within each group of curves, clustering is further conducted based on how the curves change over time. In our work, we perform the model-based clustering in a semi-supervised manner. Users' domain knowledge is integrated through intuitive interaction tools to steer the clustering process. Our dual-space method has been used to analyze particle data in combustion simulations and can also be applied to other scientific simulations involving particle trajectory analysis work.

In flow visualization, field lines are often used to convey both global and local structure and movement of the flow. One challenge is to find and classify the representative field lines. Most existing solutions follow an automatic approach that generates field lines characterizing the flow and arranges these lines into a single picture. In our work, we advocate a user-centric approach to exploring 3D vector fields. Our method allows the user to sketch 2D curves for pattern matching in 2D and field lines clustering in 3D. Specifically, a 3D field line whose view-dependent 2D projection is most similar to the user drawing will be identified and utilized to extract all similar 3D field lines. Furthermore, we employ an automatic clustering method to generate field-line templates for the user to locate subfields of interest. This semi-automatic process leverages the user's knowledge about the flow field through intuitive user interaction, resulting in a promising alternative to existing flow visualization solutions. With our sketch-based interface, the user can effectively dissect the flow field and make more structured visualization for analysis or presentation.

The iterative closest point (ICP) algorithm is an accurate and fast approach for registration between two point sets in a same scale, but it doesn't handle the case with different scales. This paper instead introduces a novel approach named the iterative closest point with bounded scale (ICPBS) algorithm which integrates a scale with boundaries into the traditional ICP algorithm. This proposed technique uses the singular value decomposition algorithm and the properties of parabola to compute the similar transformation at each iterative step, and yields more satisfying robust results than the traditional ICP method in registration between two m-D point sets with different scales. Experimental results demonstrate the presented method is robust and fast for practical use.