Generative modeling is a powerful technique that involves creating machine learning models capable of creating new data similar to the data it was trained on. Generative Adversarial Networks (GANs) are a leading approach for generative modeling. However, GAN training is known to be a notoriously difficult task. GAN convergence issues are largely caused by the supports of the real and generated distributions being disjoint. To tackle this open problem, we propose a novel GAN pre-training process that effectively aligns the supports of the generated and real data prior to applying traditional adversarial GAN training. The key component of our method, called AlignGAN, is learning a mapping between the input data distribution and a latent representation defined over a hypersphere, regularized by a One Class Classifier. This successfully encourages the generator to produce samples throughout the support of the real data, while not generating samples outside the support. We maintain support alignment through low-bandwidth noise convolutions and additional One Class regularization, leading to continued stable GAN training. We validate our approach against leading stabilization methods on three benchmark datasets, showing AlignGAN routinely produces the best results.

Human Context Recognition (HCR) from smartphone sensor data is a crucial task for Context-Aware (CA) systems, such as those targeting the healthcare and security domains. HCR models deployed in the wild are susceptible to adversarial attacks, wherein an adversary perturbs input sensor values to cause malicious mis-classifications. In this study, we demonstrate evasion attacks that can be perpetuated during model inference, particularly input perturbations that are adversarially calibrated to fool classifiers. In contrast to white-box methods that require impractical levels of system access, black-box evasion attacks merely require the ability to query the model with arbitrary inputs. Specifically, we generate adversarial perturbations using only class confidence scores, as in the Zoo attack, or only class decisions, as in the HopSkipJump (HSJ) attack that correspond with plausible scenarios of possible adversarial attacks. We empirically demonstrate that sophisticated adversarial evasion attacks can significantly impair the accuracy of HCR models, resulting in a performance drop of up to 60% in f1-score. We also propose RobustHCR, an innovative framework for demonstrating and defending against black box evasion threats using a provable defense based on a duality-based network. RobustHCR is able to make reliable predictions regardless of whether its input is under attack or not, effectively mitigating the potential negative impacts caused by adversarial attacks. Rigorous evaluation on both scripted and in-the-wild smartphone HCR datasets demonstrates that RobustHCR can significantly improve the HCR model’s robustness and protect it from possible evasion attacks while maintaining acceptable performance on "clean" inputs. In particular, an HCR model with integrated RobustHCR defenses experienced an f1-score reduction of about 3% as opposed to a reduction of over 50% for an HCR model without a defense.

Generative models are being increasingly used in science and industry applications. Unfortunately, they often perpetuate the biases present in their training sets, such as societal biases causing certain groups to be underrepresented in the data. For instance, image generators may overwhelmingly produce images of white people due to few non-white samples in their training data. It is imperative to debias generative models so they synthesize an equal number of instances for each group, while not requiring retraining of the model to avoid prohibitive expense. We thus propose a *distribution mapping module* that produces samples from a *fair noise distribution*, such that the pretrained generative model produces *semantically uniform* outputs - an equal number of instances for each group - when conditioned on these samples. This does *not* involve retraining the generator, nor does it require *any* real training data. Experiments on debiasing generators trained on popular real-world datasets show that our method outperforms existing approaches.

Digital phenotyping is the characterization of human behavior patterns based on data from digital devices such as smartphones in order to gain insights into the users’ state and especially to identify ailments. To support supervised machine learning, digital phenotyping requires gathering data from study participants’ smartphones as they live their lives. Periodically, participants are then asked to provide ground truth labels about their health status. Analyzing such complex data is challenging due to limited contextual information and imperfect health/wellness labels. We propose INteractive PHOne-o-typing VISualization (INPHOVIS), an interactive visual framework for exploratory analysis of smartphone health data to study phone-o-types.Prior visualization work has focused on mobile health data with clear semantics such as steps or heart rate data collected using dedicated health devices and wearables such as smartwatches. However, unlike smartphones which are owned by over 85 percent of the US population, wearable devices are less prevalent thus reducing the number of people from whom such data can be collected. In contrast, the “low-level” sensor data (e.g., accelerometer or GPS data) supported by INPHOVIS can be easily collected using smartphones. Data visualizations are designed to provide the essential contextualization of such data and thus help analysts discover complex relationships between observed sensor values and health predictive phone-o-types. To guide the design of INPHOVIS, we performed a hierarchical task analysis of phone-o-typing requirements with health domain experts. We then designed and implemented multiple innovative visualizations integral to INPHOVIS including stacked bar charts to show diurnal behavioral patterns, calendar views to visualize day level data along with bar charts, and correlation views to visualize important wellness predictive data. We demonstrate the usefulness of INPHOVIS with walk-throughs of use cases. We also evaluated INPHOVIS with expert feedback and received encouraging responses.

In important domains from video stream analytics to human context recognition, datasets are only partially-labeled. Worse yet, the labels are often applied sequentially, as annotators choose labels frame-by-frame or timestep-by-timestep in sequence. With labels not collected independently, this results in sequential bias in the labeling. Unfortunately, current state-of-the-art methods for partially labeled data are rendered ineffective under sequential bias. In this work, we propose a novel solution to tackling this open sequential bias problem, called DeepSPU. DeepSPU recovers missing labels by constructing a model of the sequentially biased labeling process itself. This labeling model is then learned jointly with the prediction model that infers the missing labels in an iterative training process. Further, we regulate this training using a theoretically-justified cost functions that prevent our model from converging to incorrect but low-cost solution. Our experimental studies demonstrate that DeepSPU consistently outperforms the state-of-the-art methods by over 10% on a rich variety of real-world datasets.

Open set recognition models address the real-world scenario where classes of data unobserved during training are encountered in testing after deployment. Closed set classifiers wrongly attempt to classify instances from an unknown class as belonging to one of the known classes from the training set, which reduces the model’s accuracy. Ideally, these unknown instances should be recognized as such, while known instances should continue to be accurately classified. Unfortunately, state-of-the-art open set methods solve this problem by making restrictive assumptions on the variance and/or boundedness of the distributions of known classes. In this paper, we propose a novel method, Variational Open-Set Recognition (VOSR) that eliminates these assumptions. VOSR incorporates a closed set classifier, an unknown detector, and a novel Structured Gaussian Mixture Variational Autoencoder (SGM-VAE) that guarantees separable class distributions with known variances in its la-tent space. Further, by encouraging a large distance between class-specific distributions, VOSR increases the likelihood that instances from unknown classes lie in low-probability regions and thus are more readily identifiable. In rigorous evaluation, we demonstrate that VOSR outperforms state-of-the-art open set classifiers with up to a 14% F1 score increase in identifying instances from unknown classes in multiple image classification and human activity recognition datasets.

Human activity recognition (HAR), the task of predicting the activities performed by an individual using mobile sensor data, is an active and important area of research. Unfortunately, it is very costly to collect the data required to train robust HAR classifiers. To tackle this issue, there has been an increasing focus on generating synthetic HAR data for augmentation purposes. The state-of-the-art generative HAR approaches utilize Generative Adversarial Networks (GANs) to produce realistic synthetic HAR data. However, these solutions can not generate personalized data that matches the behavior of particular users, limiting their potential use cases. This is particularly problematic in the mobile health domain, where the target users are often elderly or disabled and are thus likely to have activity signals that are unique from the general population. To overcome this drawback, we propose a novel controllable GAN solution Control-HAR-GAN. Our approach learns user and activity signals independently, and when generating synthetic instances, practitioners can specify both the activity to be generated as well as the user the data should match. This has the added benefit in that our model supports novel user-activity pairs by generating examples that match the data that would have been recorded by a particular user if they had performed the target activity, even if the user never performed that activity during the data collection process. We show that our model outperforms the existing HAR GAN approach in generating observed user-activity pairs by up to 10%. Additionally, our approach can also perform the task of novel user-activity pair generation, which is impossible for existing approaches.

Human Context Recognition (HCR) is an important field at the intersection of personalized healthcare and ubiquitous computing that aims to describe the current state of a person given mobile sensor data. Previous work has demonstrated the ability to detect various ailments through context patterns and deviations from those patterns, illustrating an important application of HCR. Contexts captured by mobile sensor data are often associated with high intra-subject and inter-subject variability, which hinders the process of effectively learning representations for each class. Obtaining a sufficient quantity of labeled context data for reliable predictions becomes prohibitively expensive as the number of users and targeted contexts increases. Few-Shot Classification is a family of methods wherein models are trained with limited amounts of labeled data such that their latent representations can be used for transferring information from one context to another. In this paper, we provide the first comprehensive study of state-of-the-art Few-Shot classifiers on HCR data. We find that methods that fine-tune classifiers using a general feature extractor on limited labeled samples outperform more sophisticated Few Shot techniques in the HCR setting by up to 50%.

The quality of the features engineered to represent data during dimension reduction greatly impacts the performance of machine learning. Autoencoders are a popular deep-learning approach for generating lower-dimensional representations of complex, high-dimensional data. Unfortunately, features derived from Autoencoders do not preserve the local geometry of data. Thus, relationships between data may be lost during the featurization process, leading to lower-quality representations. To address this, we propose Deep Maximum Variance Unfolding (DMVU), a deep learning architecture for manifold-preserving dimension reduction. DMVU is a deep-learning extension of Maximum Variance Unfolding (MVU), a classic machine learning approach for dimension reduction. DMVU retains the local geometry preserving characteristics of MVU, while leveraging deep learning’s representational capacity and ability to manage large, high-dimensional datasets. This is achieved by DMVU’s novel loss function, which encourages the distance between neighboring points to be preserved while encouraging overall variance between data points. Our experimental study demonstrates that DMVU consistently outperforms state-of-the-art methods using the standard metrics of continuity and trustworthiness on 4 benchmark datasets.

The recognition of complex activities such as "having dinner" or "cooking" from wearable sensor data is an important problem in various healthcare, security and context-aware mobile and ubiquitous computing applications. In contrast to simple activities such as walking that involve single, indivisible repeated actions, recognizing complex activities such as "having dinner" is a harder sub-problem that may be composed of multiple interleaved or concurrent simple activities with different orderings each time. Most of prior work has focused on recognizing simple activities, used hand-crafted features, or did not perform classification using a state-of-the-art neural networks model. In this paper, we propose CARTMAN, a complex activity recognition method that uses Latent Dirichlet allocation (LDA) topic models to generate smartphone sensor features that capture the latent representation of complex activities. These LDA features are then classified using a DeepConvLSTM neural network with self-attention. DeepConvLSTM auto-learns the spatio-temporal features from the sensor data while the self-attention layer identifies and focuses on the predictive points within the time-series sensor data. Our CARTMAN approach outperforms the current state-of-the-art complex activity models and baseline models by 6-23% in macro and weighted F1-scores.

Human Bio-Behavioral Rhythms (HBRs) such as sleep-wake cycles (Circadian Rhythms), and the degree of regularity of sleep and physical activity have important health ramifications. Ubiquitous devices such as smartphones can sense HBRs by continuously analyzing data gathered passively by built-in sensors to discover important clues about the degree of regularity and disruptions in behavioral patterns. As human behavior is complex and smartphone data is voluminous with many channels (sensor types), it can be challenging to make meaningful observations, detect unhealthy HBR deviations and most importantly pin-point the causes of disruptions. Prior work has largely utilized computational methods such as machine and deep learning approaches, which while accurate, are often not explainable and present few actionable insights on HBR patterns or causes. To assist analysts in the discovery and understanding of HBR patterns, disruptions and causes, we propose ARGUS, an interactive visual analytics framework. As a foundation of ARGUS, we design an intuitive Rhythm Deviation Score (RDS) that analyzes users’ smartphone sensor data, extracts underlying twenty-four-hour rhythms and quantifies their degree of irregularity.​ This score is then visualized using a glyph that makes it easy to recognize disruptions in the regularity of HBRs. ARGUS also facilitates deeper HBR insights and understanding of causes by linking multiple visualization panes that are overlaid with objective sensor information such as geo-locations and phone state (screen locked, charging), and user-provided or smartphone-inferred ground truth information. This array of visualization overlays in ARGUS enables​ analysts to gain a more comprehensive picture of HBRs, behavioral patterns and deviations from regularity. The design of ARGUS was guided by a goal and task analysis study involving an expert versed in HBR and smartphone sensing. To demonstrate its utility and generalizability, two different datasets were explored using ARGUS and our use cases and designs were strongly validated in evaluation sessions with expert and non-expert users.

In big data applications from digital health to assisted living smart systems, only a fraction of data instances used for training classifiers t end to be labeled. One important subfield of weakly labeled learning, called Positive Unlabeled (PU) learning, does not require a completely labeled dataset in order to train a strong classifier. This is crucial as in many domains it is expensive or impossible to obtain a completely labeled dataset. While prior PU work assumed that unlabeled instances occurred with a random uniform distribution, we observe that labeled (and unlabeled) data tends to occur in long contiguous sequences (or bursts) due the prevalent burst labeling behavior by human annotators. Burst labeling leads to a sequential bias in PU data not addressed by state-of-the-art methods. To tackle this open problem of learning under sequential bias, we propose BurstPU, the first framework for training a classifier on sequentially labeled PU data. BurstPU addresses the challenge that two interdependent models must be learned, namely, the classification model and the labeling likelihood model, with the later predicting the likelihood that a given instance is labeled. The labeling likelihood model is then needed during the training of the classification model to account for the bias in the labeling process. Our experimental study demonstrates that BurstPU consistently outperforms all state-of-the-art PU methods on a rich variety of diverse real-world datasets, and can learn from fewer labeled instances compared to state-of-art PU methods.

Current research in passive health monitoring utilizes machine learning methods to infer users' symptoms and health status from smartphone-sensed data, which can be gathered on a large scale. However, reasoning about smartphone-sensed health behaviors should engage health experts who may not want to solely rely on computational approaches as they provide limited insights. We designed and proposed a visualization framework for the INTeractive Observation of Smartphone-Inferred Symptoms (INTOSIS), that supports contextualization of symptomatic days by presenting a holistic picture of complex smartphone data for analysts to find c oncerning b ehavior patterns. For instance, while sedentary behavior caused by the flu is concerning, sedentary behaviors on holidays are non-concerning. INTOSIS visualizes multiple smartphone sensor data channels such as geo-location, app usage and screen usage. It uses visual metaphors to effectively represent the data to help analysts derive important human-understandable spatio-temporal contexts and assign health consequences with corresponding semantic labels. INTOSIS provides timeline visualizations of contextual clues such as screen and app usage at night that support an analyst in being able to reason about and then extract plausible explanations for the occurrence of certain symptoms, such as, sleep problems. We validate INTOSIS with intuitive use cases, using a real-world smartphone-sensed dataset, along with expert evaluation.