The learning dynamics of modern neural networks remain an open problem in deep learning. The Neural Tangent Kernel (NTK) offers an elegant description of training dynamics in the infinite‑width limit, yet its classical formulation assumes a static data set. Modern model training practice departs from this strong assumption through the use of on‑the‑fly data augmentations (e.g. additive noise). In this work, we conduct an NTK-driven analysis of how data transformations affect a neural net's evolution in the function space. Our theoretical contributions characterize how repeated Gaussian perturbations from NTK-derived covariances can steer neural-net optimizations toward user‑specified behavior. These theoretical insights are empirically validated by controlled experiments. Taken together, our results lay the foundation for a promising future research direction that transforms the NTK from a descriptive to a prescriptive tool, enabling control of neural net training trajectories and behavior of inference generalization with grounded interventions.

An interesting behavior in large language models (LLMs) is prompt sensitivity. When provided with different but semantically equivalent versions of the same prompt, models may produce very different distributions of answers. This suggests that the uncertainty reflected in a model's output distribution for one prompt may not reflect the model's uncertainty about the meaning of the prompt. We model prompt sensitivity as a type of generalization error, and show that sampling across the semantic concept space with paraphrasing perturbations improves uncertainty calibration without compromising accuracy. Additionally, we introduce a new metric for uncertainty decomposition in black-box LLMs that improves upon entropy-based decomposition by modeling semantic continuities in natural language generation. We show that this decomposition metric can be used to quantify how much LLM uncertainty is attributed to prompt sensitivity. Our work introduces a new way to improve uncertainty calibration in prompt-sensitive language models, and provides evidence that some LLMs fail to exhibit consistent general reasoning about the meanings of their inputs.

Developments in deep neural nets have trended towards increasingly larger overparameterized architectures, resulting in lengthy training sessions with ever more elusive training dynamics. Thus, ensuring these models learn accurate generalizable representations of data efficiently is challenging. Previous works have developed specialized techniques from data-pruning, architecture selection, pseudo-label generation, bias identification, or label refurbishment to improve downstream training. Problematically, most methods require prohibitively expensive iterative model training. In this paper, we demonstrate that we can exploit the recent neural tangent kernel (NTK) theory for understanding and improving model training behavior before ever training a model. First, we show a powerful signal derived from the NTK theory can be computed remarkably fast. We then leverage this signal for the design of a unified suite of surprisingly effective tools for the four important tasks of architecture selection, pseudo-label verification, bias identification, and label refurbishment, all requiring zero model training.

Many time series classification tasks, where labels vary over time, are affected by label noise that also varies over time. Such noise can cause label quality to improve, worsen, or periodically change over time. We first propose and formalize temporal label noise, an unstudied problem for sequential classification of time series. In this setting, multiple labels are recorded over time while being corrupted by a time-dependent noise function. We first demonstrate the importance of modeling the temporal nature of the label noise function and how existing methods will consistently underperform. We then propose methods to train noise-tolerant classifiers by estimating the temporal label noise function directly from data. We show that our methods lead to state-of-the-art performance under diverse types of temporal label noise on real-world datasets.

Over recent years, many sophisticated methods for the critical problem of domain adaptation (DA) have been proposed in the literature. Unfortunately, there is no single DA model that always outperforms others regardless of datasets and setups. Motivated by this, we investigate the effect of ensembles of (imperfect) classic DA models – each pre-trained on distinct folds of data – on the ability to learn a simple yet strong DA model for solving a new target problem. We show that a simple fusion of approximately statistically independent models, modeled with a simple linear ensemble model under semi-supervision (1-/3-shot examples), boosts the performance over each individual DA model. We demonstrate that, when taken together, the rich variety of ensembled models succeeds to better cover the feature space for domain adaptation, achieving improvements over state-of-the-art semi-supervised single-source performance on benchmark data sets: DomainNet (+8.8% 1-shot, +3.1% 3-shot) and Office-31 (+15.8% 1-shot, +8.4% 3-shot). We release our code for reproducibility.

Foundation models trained on large-scale de-identified electronic health records (EHRs) hold promise for clinical applications. However, their capacity to memorize patient information raises important privacy concerns. In this work, we introduce a suite of black-box evaluation tests to assess privacy-related memorization risks in foundation models trained on structured EHR data. Our framework includes methods for probing memorization at both the embedding and generative levels, and aims to distinguish between model generalization and harmful memorization in clinically relevant settings. We contextualize memorization in terms of its potential to compromise patient privacy, particularly for vulnerable subgroups. We validate our approach on a publicly available EHR foundation model and release an open-source toolkit to facilitate reproducible and collaborative privacy assessments in healthcare AI.

There are increasingly many large language models (LLMs) available to the public. While these LLMs have exhibited impressive abilities on a variety of task, any individual LLM in particular may do well on some tasks and worse on others. Additionally, the performance of these models is heavily dependent on the choice of prompt template used. For instance, they exhibit sensitivity to the few shot examples chosen or brittleness to the wording of instructions. Moreover, a prompt template that makes a model perform well for one input may not be the optimal template for another input. This necessitates an approach for adaptively selecting LLM and prompt template pairs for each input. Recent work has shown that the accuracy of LLM's responses is correlated with the LLM's confidence in the response. Thus, a natural choice for selecting which model and prompt template to use is to select the pair that is most confident in its response. However, existing confidence metrics are expensive to calculate - necessitating multiple calls to each LLm and prompt pair. We thus propose an approach to predict the confidence of each pair using an auxiliary regression model that is inexpensive to run. Using this auxiliary model, we select the LLM and prompt template with the highest predicted confidence for a given input. Results on a range of benchmark datasets show that our confidence-based instance-level prompt search method consistently improves the performance of LLMs.

Multi-task learning (MTL) is essential for real-world applications that handle multiple tasks simultaneously, such as selfdriving cars. MTL methods improve the performance of all tasks by utilizing information across tasks to learn a robust shared representation. However, acquiring sufficient labeled data tends to be extremely expensive, especially when having to support many tasks. Recently, Knowledge Amalgamation (KA) has emerged as an effective strategy for addressing the lack of labels by instead learning directly from pretrained models (teachers). KA learns one unified multi-task student that masters all tasks across all teachers. Existing KA for MTL works are limited to teachers with identical architectures, and thus propose layer-to-layer based approaches. Unfortunately, in practice, teachers may have heterogeneous architectures; their layers may not be aligned and their dimensionalities or scales may be incompatible. Amalgamating multi-task teachers with heterogeneous architectures remains an open problem. For this, we design Versatile Common Feature Consolidator (VENUS), the first solution to this problem. VENUS fuses knowledge from the shared representations of each teacher into one unified generalized representation for all tasks. Specifically, we design the Feature Consolidator network that leverages an array of teacher-specific trainable adaptors. These adaptors enable the student to learn from multiple teachers, even if they have incompatible learned representations. We demonstrate that VENUS outperforms five alternative methods on numerous benchmark datasets across a broad spectrum of experiments.

Vision-language models, like CLIP (Contrastive Language Image Pretraining), are becoming increasingly popular for a wide range of multimodal retrieval tasks. However, prior work has shown that large language and deep vision models can learn historical biases contained in their training sets, leading to perpetuation of stereotypes and potential downstream harm. In this work, we conduct a systematic analysis of the social biases that are present in CLIP, with a focus on the interaction between image and text modalities. We first propose a taxonomy of social biases called So-B-It, which contains 374 words categorized across ten types of bias. Each type can lead to societal harm if associated with a particular demographic group. Using this taxonomy, we examine images retrieved by CLIP from a facial image dataset using each word as part of a prompt. We find that CLIP frequently displays undesirable associations between harmful words and specific demographic groups, such as retrieving mostly pictures of Middle Eastern men when asked to retrieve images of a "terrorist". Finally, we conduct an analysis of the source of such biases, by showing that the same harmful stereotypes are also present in a large image-text dataset used to train CLIP models for examples of biases that we find. Our findings highlight the importance of evaluating and addressing bias in vision-language models, and suggest the need for transparency and fairness-aware curation of large pre-training datasets.

Generative Adversarial Networks (GANs) are a generative framework with a notorious reputation for instability. Despite significant work in attempting to improve stability, training remains extremely difficult in practice. Nearly all GAN optimization methods are built on either simultaneous (Sim-GDA) or alternating (Alt-GDA) gradient descent-ascent, where the generator and discriminator are updated either at the same time iteratively or in a fixed pattern. Unfortunately, neither Sim-GDA nor Alt-GDA have any strongly convergent properties, nor are they Lyapunov-stable. In this paper, we prove for simple GANs, for which training had been proven non-convergent under Sim-GDA and Alt-GDA, that our newly introduced training method is Lyapunov-stable. We then design a novel oracle-guided GDA training strategy called Dynamic-GDA that leverages generalized analogs of the properties exhibited in the simple case. We also prove that in contrast to Sim/Alt-GDA, GANs with Dynamic-GDA achieve Lyapunov-stable training with non-infinitesimal learning rates. Empirically, we show Dynamic-GDA improves convergence orthogonally to common stabilizing techniques on 8 classes of GAN models and 7 different data sets.

Vision-language (VL) embedding models have been shown to encode biases present in their training data, such as societal biases that prescribe negative characteristics to members of various racial and gender identities. Due to their wide-spread adoption for various tasks ranging from few-shot classification to text-guided image generation, debiasing VL models is crucial. Debiasing approaches that fine-tune the VL model often suffer from catastrophic forgetting. On the other hand, fine-tuning-free methods typically utilize a ``one-size-fits-all" approach that assumes that correlation with the spurious attribute can be explained using a single linear direction across all possible inputs. In this work, we propose a nonlinear, fine-tuning-free approach for VL embedding model debiasing that tailors the debiasing operation to each unique input. This allows for a more flexible debiasing approach. Additionally, we do not require knowledge of the set of inputs a priori to inference time, making our method more appropriate for online tasks such as retrieval and text guided image generation.

Multi-label classification (MLC), which assigns multiple labels to each instance, is crucial to domains from computer vision to text mining. Conventional methods for MLC require huge amounts of labeled data to capture complex dependencies between labels. However, such labeled datasets are expensive, or even impossible, to acquire. Worse yet, these pre-trained MLC models can only be used for the particular label set covered in the training data. Despite this severe limitation, few methods exist for expanding the set of labels predicted by pre-trained models. Instead, we acquire vast amounts of new labeled data and retrain a new model from scratch. Here, we propose combining the knowledge from multiple pre-trained models (teachers) to train a new student model that covers the union of the labels predicted by this set of teachers. This student supports a broader label set than any one of its teachers without using labeled data. We call this new problem knowledge amalgamation for multi-label classification. Our new method, Adaptive KNowledge Transfer (ANT), trains a student by learning from each teacher’s partial knowledge of label dependencies to infer the global dependencies between all labels across the teachers. We show that ANT succeeds in unifying label dependencies among teachers, outperforming five state-of-the-art methods on eight real-world datasets.

Generative modeling is a powerful technique that involves creating machine learning models capable of creating new data similar to the data it was trained on. Generative Adversarial Networks (GANs) are a leading approach for generative modeling. However, GAN training is known to be a notoriously difficult task. GAN convergence issues are largely caused by the supports of the real and generated distributions being disjoint. To tackle this open problem, we propose a novel GAN pre-training process that effectively aligns the supports of the generated and real data prior to applying traditional adversarial GAN training. The key component of our method, called AlignGAN, is learning a mapping between the input data distribution and a latent representation defined over a hypersphere, regularized by a One Class Classifier. This successfully encourages the generator to produce samples throughout the support of the real data, while not generating samples outside the support. We maintain support alignment through low-bandwidth noise convolutions and additional One Class regularization, leading to continued stable GAN training. We validate our approach against leading stabilization methods on three benchmark datasets, showing AlignGAN routinely produces the best results.

Human Context Recognition (HCR) from smartphone sensor data is a crucial task for Context-Aware (CA) systems, such as those targeting the healthcare and security domains. HCR models deployed in the wild are susceptible to adversarial attacks, wherein an adversary perturbs input sensor values to cause malicious mis-classifications. In this study, we demonstrate evasion attacks that can be perpetuated during model inference, particularly input perturbations that are adversarially calibrated to fool classifiers. In contrast to white-box methods that require impractical levels of system access, black-box evasion attacks merely require the ability to query the model with arbitrary inputs. Specifically, we generate adversarial perturbations using only class confidence scores, as in the Zoo attack, or only class decisions, as in the HopSkipJump (HSJ) attack that correspond with plausible scenarios of possible adversarial attacks. We empirically demonstrate that sophisticated adversarial evasion attacks can significantly impair the accuracy of HCR models, resulting in a performance drop of up to 60% in f1-score. We also propose RobustHCR, an innovative framework for demonstrating and defending against black box evasion threats using a provable defense based on a duality-based network. RobustHCR is able to make reliable predictions regardless of whether its input is under attack or not, effectively mitigating the potential negative impacts caused by adversarial attacks. Rigorous evaluation on both scripted and in-the-wild smartphone HCR datasets demonstrates that RobustHCR can significantly improve the HCR model’s robustness and protect it from possible evasion attacks while maintaining acceptable performance on "clean" inputs. In particular, an HCR model with integrated RobustHCR defenses experienced an f1-score reduction of about 3% as opposed to a reduction of over 50% for an HCR model without a defense.

Generative models are being increasingly used in science and industry applications. Unfortunately, they often perpetuate the biases present in their training sets, such as societal biases causing certain groups to be underrepresented in the data. For instance, image generators may overwhelmingly produce images of white people due to few non-white samples in their training data. It is imperative to debias generative models so they synthesize an equal number of instances for each group, while not requiring retraining of the model to avoid prohibitive expense. We thus propose a *distribution mapping module* that produces samples from a *fair noise distribution*, such that the pretrained generative model produces *semantically uniform* outputs - an equal number of instances for each group - when conditioned on these samples. This does *not* involve retraining the generator, nor does it require *any* real training data. Experiments on debiasing generators trained on popular real-world datasets show that our method outperforms existing approaches.

Digital phenotyping is the characterization of human behavior patterns based on data from digital devices such as smartphones in order to gain insights into the users’ state and especially to identify ailments. To support supervised machine learning, digital phenotyping requires gathering data from study participants’ smartphones as they live their lives. Periodically, participants are then asked to provide ground truth labels about their health status. Analyzing such complex data is challenging due to limited contextual information and imperfect health/wellness labels. We propose INteractive PHOne-o-typing VISualization (INPHOVIS), an interactive visual framework for exploratory analysis of smartphone health data to study phone-o-types.Prior visualization work has focused on mobile health data with clear semantics such as steps or heart rate data collected using dedicated health devices and wearables such as smartwatches. However, unlike smartphones which are owned by over 85 percent of the US population, wearable devices are less prevalent thus reducing the number of people from whom such data can be collected. In contrast, the “low-level” sensor data (e.g., accelerometer or GPS data) supported by INPHOVIS can be easily collected using smartphones. Data visualizations are designed to provide the essential contextualization of such data and thus help analysts discover complex relationships between observed sensor values and health predictive phone-o-types. To guide the design of INPHOVIS, we performed a hierarchical task analysis of phone-o-typing requirements with health domain experts. We then designed and implemented multiple innovative visualizations integral to INPHOVIS including stacked bar charts to show diurnal behavioral patterns, calendar views to visualize day level data along with bar charts, and correlation views to visualize important wellness predictive data. We demonstrate the usefulness of INPHOVIS with walk-throughs of use cases. We also evaluated INPHOVIS with expert feedback and received encouraging responses.

Positive-Unlabeled (PU) learning methods train a classifier to distinguish between the positive and negative classes given only positive and unlabeled data. While traditional PU methods require the labeled positive samples to be an unbiased sample of the positive distribution, in practice the labeled sample is often a biased draw from the true distribution. Prior work shows that if we know the likelihood that each positive instance will be selected for labeling, referred to as the propensity score, then the biased sample can be used for PU learning. Unfortunately, no prior work has been proposed an inference strategy for which the propensity score is identifiable. In this work, we propose two sets of assumptions under which the propensity score can be uniquely determined: one in which no assumption is made on the functional form of the propensity score (requiring assumptions on the data distribution), and the second which loosens the data assumptions while assuming a functional form for the propensity score. We then propose inference strategies for each case. Our empirical study shows that our approach significantly outperforms the state-of-the-art propensity estimation methods on a rich variety of benchmark datasets.

Human activity recognition (HAR) is the process of using mobile sensor data to determine the physical activities performed by individuals. HAR is the backbone of many mobile healthcare applications, such as passive health monitoring systems, early diagnosing systems, and fall detection systems. Effective HAR models rely on deep learning architectures and big data in order to accurately classify activities. Unfortunately, HAR datasets are expensive to collect, are often mislabeled, and have large class imbalances. State-of-the-art approaches to address these challenges utilize Generative Adversarial Networks (GANs) for generating additional synthetic data along with their labels. Problematically, these HAR GANs only synthesize continuous features — features that are represented by real numbers — recorded from gyroscopes, accelerometers, and other sensors that produce continuous data. This is limiting since mobile sensor data commonly has discrete features that provide additional context such as device location and the time-of-day, which have been shown to substantially improve HAR classification. Hence, we studied Conditional Tabular Generative Adversarial Networks (CTGANs) for data generation to synthesize mobile sensor data containing both continuous and discrete features, a task never been done by state-of-the-art approaches. We show HAR-CTGANs generate data with greater realism resulting in allowing better downstream performance in HAR models, and when state-of-the-art models were modified with HAR-CTGAN characteristics, downstream performance also improves.

Corpuses of unstructured textual data, such as text messages between individuals, are often predictive of medical issues such as depression. The text data usually used in healthcare applications has high value and great variety, but is typically small in volume. Generating labeled unstructured text data is important to improve models by augmenting these small datasets, as well as to facilitate anonymization. While methods for labeled data generation exist, not all of them generalize well to small datasets. In this work, we thus perform a much needed systematic comparison of conditional text generation models that are promising for small datasets due to their unified architectures. We identify and implement a family of nine conditional sequence generative adversarial networks for text generation, which we collectively refer to as cSeqGAN models. These models are characterized along two orthogonal design dimensions: weighting strategies and feedback mechanisms. We conduct a comparative study evaluating the generation ability of the nine cSeqGAN models on three diverse text datasets with depression and sentiment labels. To assess the quality and realism of the generated text, we use standard machine learning metrics as well as human assessment via a user study. While the unconditioned models produced predictive text, the cSeqGAN models produced more realistic text. Our comparative study lays a solid foundation and provides important insights for further text generation research, particularly for the small datasets common within the healthcare domain.

Recurrent Classifier Chains (RCCs) are a leading approach for multi-label classification as they directly model the interdependencies between classes. Unfortunately, existing RCCs assume that every training instance is completely labeled with all its ground truth classes. In practice often only a subset of an instance's labels are annotated, while the annotations for other classes aremissing. RCCs fail in this missing label scenario, predicting many false negatives and potentially missing important classes. In this work, we propose Robust-RCC, the first strategy for tackling this open problem of RCCs failing formulti-label missing-label data. Robust-RCC is a new type of deep recurrent classifier chain empowered to model inter-class relationships essential for predicting thecomplete label set most likely to match the ground truth. The key to Robust-RCC is the design of the Multi Incomplete Label Risk (MILR) function, which we prove to be equal in expectation to the true risk of the ground truth full label set despite being computed from incompletely labeled data. Our experimental study demonstrates that Robust-RCC consistently beats six state-of-of-the-art methods by as much as 30% in predicting the true labels.

Early classification algorithms help users react faster to their machine learning model's predictions. Early warning systems in hospitals, for example, let clinicians improve their patients' outcomes by accurately predicting infections. While early classification systems are advancing rapidly, a major gap remains: existing systems do not consider irregular time series, which have uneven and often-long gaps between their observations. Such series are notoriously pervasive in impactful domains like healthcare. We bridge this gap and study early classification of irregular time series, a new setting for early classifiers that opens doors to more real-world problems. Our solution, Stop&Hop, uses a continuous-time recurrent network to model ongoing irregular time series in real time, while an irregularity-aware halting policy, trained with reinforcement learning, predicts when to stop and classify the streaming series. By taking real-valued step sizes, the halting policy flexibly decides exactly when to stop ongoing series in real time. This way, Stop&Hop seamlessly integrates information contained in the timing of observations, a new and vital source for early classification in this setting, with the time series values to provide early classifications for irregular time series. Using four synthetic and three real-world datasets, we demonstrate that Stop&Hop consistently makes earlier and more-accurate predictions than state-of-the-art alternatives adapted to this new problem. Our code is publicly available at https://github.com/thartvigsen/StopAndHop.

In important domains from video stream analytics to human context recognition, datasets are only partially-labeled. Worse yet, the labels are often applied sequentially, as annotators choose labels frame-by-frame or timestep-by-timestep in sequence. With labels not collected independently, this results in sequential bias in the labeling. Unfortunately, current state-of-the-art methods for partially labeled data are rendered ineffective under sequential bias. In this work, we propose a novel solution to tackling this open sequential bias problem, called DeepSPU. DeepSPU recovers missing labels by constructing a model of the sequentially biased labeling process itself. This labeling model is then learned jointly with the prediction model that infers the missing labels in an iterative training process. Further, we regulate this training using a theoretically-justified cost functions that prevent our model from converging to incorrect but low-cost solution. Our experimental studies demonstrate that DeepSPU consistently outperforms the state-of-the-art methods by over 10% on a rich variety of real-world datasets.

Ranking evaluation metrics play an important role in information retrieval, providing optimization objectives during development and means of assessment of deployed performance. Recently, fairness of rankings has been recognized as crucial, especially as automated systems are increasingly used for high impact decisions. While numerous fairness metrics have been proposed, a comparative analysis to understand their interrelationships is lacking. Even for fundamental statistical parity metrics which measure group advantage, it remains unclear whether metrics measure the same phenomena, or when one metric may produce different results than another. To address these open questions, we formulate a conceptual framework for analytical comparison of metrics. We prove that under reasonable assumptions, popular metrics in the literature exhibit the same behavior and that optimizing for one optimizes for all. However, our analysis also shows that the metrics vary in the degree of unfairness measured, in particular when one group has a strong majority. Based on this analysis, we design a practical statistical test to identify whether observed data is likely to exhibit predictable group bias. We provide a set of recommendations for practitioners to guide the choice of an appropriate fairness metric.

Open set recognition models address the real-world scenario where classes of data unobserved during training are encountered in testing after deployment. Closed set classifiers wrongly attempt to classify instances from an unknown class as belonging to one of the known classes from the training set, which reduces the model’s accuracy. Ideally, these unknown instances should be recognized as such, while known instances should continue to be accurately classified. Unfortunately, state-of-the-art open set methods solve this problem by making restrictive assumptions on the variance and/or boundedness of the distributions of known classes. In this paper, we propose a novel method, Variational Open-Set Recognition (VOSR) that eliminates these assumptions. VOSR incorporates a closed set classifier, an unknown detector, and a novel Structured Gaussian Mixture Variational Autoencoder (SGM-VAE) that guarantees separable class distributions with known variances in its la-tent space. Further, by encouraging a large distance between class-specific distributions, VOSR increases the likelihood that instances from unknown classes lie in low-probability regions and thus are more readily identifiable. In rigorous evaluation, we demonstrate that VOSR outperforms state-of-the-art open set classifiers with up to a 14% F1 score increase in identifying instances from unknown classes in multiple image classification and human activity recognition datasets.

Human activity recognition (HAR), the task of predicting the activities performed by an individual using mobile sensor data, is an active and important area of research. Unfortunately, it is very costly to collect the data required to train robust HAR classifiers. To tackle this issue, there has been an increasing focus on generating synthetic HAR data for augmentation purposes. The state-of-the-art generative HAR approaches utilize Generative Adversarial Networks (GANs) to produce realistic synthetic HAR data. However, these solutions can not generate personalized data that matches the behavior of particular users, limiting their potential use cases. This is particularly problematic in the mobile health domain, where the target users are often elderly or disabled and are thus likely to have activity signals that are unique from the general population. To overcome this drawback, we propose a novel controllable GAN solution Control-HAR-GAN. Our approach learns user and activity signals independently, and when generating synthetic instances, practitioners can specify both the activity to be generated as well as the user the data should match. This has the added benefit in that our model supports novel user-activity pairs by generating examples that match the data that would have been recorded by a particular user if they had performed the target activity, even if the user never performed that activity during the data collection process. We show that our model outperforms the existing HAR GAN approach in generating observed user-activity pairs by up to 10%. Additionally, our approach can also perform the task of novel user-activity pair generation, which is impossible for existing approaches.

Human Context Recognition (HCR) is an important field at the intersection of personalized healthcare and ubiquitous computing that aims to describe the current state of a person given mobile sensor data. Previous work has demonstrated the ability to detect various ailments through context patterns and deviations from those patterns, illustrating an important application of HCR. Contexts captured by mobile sensor data are often associated with high intra-subject and inter-subject variability, which hinders the process of effectively learning representations for each class. Obtaining a sufficient quantity of labeled context data for reliable predictions becomes prohibitively expensive as the number of users and targeted contexts increases. Few-Shot Classification is a family of methods wherein models are trained with limited amounts of labeled data such that their latent representations can be used for transferring information from one context to another. In this paper, we provide the first comprehensive study of state-of-the-art Few-Shot classifiers on HCR data. We find that methods that fine-tune classifiers using a general feature extractor on limited labeled samples outperform more sophisticated Few Shot techniques in the HCR setting by up to 50%.

The quality of the features engineered to represent data during dimension reduction greatly impacts the performance of machine learning. Autoencoders are a popular deep-learning approach for generating lower-dimensional representations of complex, high-dimensional data. Unfortunately, features derived from Autoencoders do not preserve the local geometry of data. Thus, relationships between data may be lost during the featurization process, leading to lower-quality representations. To address this, we propose Deep Maximum Variance Unfolding (DMVU), a deep learning architecture for manifold-preserving dimension reduction. DMVU is a deep-learning extension of Maximum Variance Unfolding (MVU), a classic machine learning approach for dimension reduction. DMVU retains the local geometry preserving characteristics of MVU, while leveraging deep learning’s representational capacity and ability to manage large, high-dimensional datasets. This is achieved by DMVU’s novel loss function, which encourages the distance between neighboring points to be preserved while encouraging overall variance between data points. Our experimental study demonstrates that DMVU consistently outperforms state-of-the-art methods using the standard metrics of continuity and trustworthiness on 4 benchmark datasets.

Exact multi-label classification is the task of assigning each datapoint a set of class labels such that the assigned set exactly matches the ground truth. Optimizing for exact multi-label classification is important in domains where missing a single label can be especially costly, such as in object detection for autonomous vehicles or symptom classification for disease diagnosis. Recurrent Classifier Chains (RCCs), a recurrent neural network extension of ensemble-based classifier chains, are the state-of-the-art exact multi-label classification method for maximizing subset accuracy. However, RCCs iteratively predict classes with an unprincipled ordering, and therefore indiscriminately condition class probabilities. These disadvantages make RCCs prone to predicting inaccurate label sets. In this work we propose Recurrent Bayesian Classifier Chains (RBCCs), which learn a Bayesian network of class dependencies and leverage this network in order to condition the prediction of child nodes only on their parents. By conditioning predictions in this way, we perform principled and non-noisy class prediction. We demonstrate the effectiveness of our RBCC method on a variety of real-world multi-label datasets, where we routinely outperform the state of the art methods for exact multi-label classification.

The recognition of complex activities such as "having dinner" or "cooking" from wearable sensor data is an important problem in various healthcare, security and context-aware mobile and ubiquitous computing applications. In contrast to simple activities such as walking that involve single, indivisible repeated actions, recognizing complex activities such as "having dinner" is a harder sub-problem that may be composed of multiple interleaved or concurrent simple activities with different orderings each time. Most of prior work has focused on recognizing simple activities, used hand-crafted features, or did not perform classification using a state-of-the-art neural networks model. In this paper, we propose CARTMAN, a complex activity recognition method that uses Latent Dirichlet allocation (LDA) topic models to generate smartphone sensor features that capture the latent representation of complex activities. These LDA features are then classified using a DeepConvLSTM neural network with self-attention. DeepConvLSTM auto-learns the spatio-temporal features from the sensor data while the self-attention layer identifies and focuses on the predictive points within the time-series sensor data. Our CARTMAN approach outperforms the current state-of-the-art complex activity models and baseline models by 6-23% in macro and weighted F1-scores.

Human Bio-Behavioral Rhythms (HBRs) such as sleep-wake cycles (Circadian Rhythms), and the degree of regularity of sleep and physical activity have important health ramifications. Ubiquitous devices such as smartphones can sense HBRs by continuously analyzing data gathered passively by built-in sensors to discover important clues about the degree of regularity and disruptions in behavioral patterns. As human behavior is complex and smartphone data is voluminous with many channels (sensor types), it can be challenging to make meaningful observations, detect unhealthy HBR deviations and most importantly pin-point the causes of disruptions. Prior work has largely utilized computational methods such as machine and deep learning approaches, which while accurate, are often not explainable and present few actionable insights on HBR patterns or causes. To assist analysts in the discovery and understanding of HBR patterns, disruptions and causes, we propose ARGUS, an interactive visual analytics framework. As a foundation of ARGUS, we design an intuitive Rhythm Deviation Score (RDS) that analyzes users’ smartphone sensor data, extracts underlying twenty-four-hour rhythms and quantifies their degree of irregularity.​ This score is then visualized using a glyph that makes it easy to recognize disruptions in the regularity of HBRs. ARGUS also facilitates deeper HBR insights and understanding of causes by linking multiple visualization panes that are overlaid with objective sensor information such as geo-locations and phone state (screen locked, charging), and user-provided or smartphone-inferred ground truth information. This array of visualization overlays in ARGUS enables​ analysts to gain a more comprehensive picture of HBRs, behavioral patterns and deviations from regularity. The design of ARGUS was guided by a goal and task analysis study involving an expert versed in HBR and smartphone sensing. To demonstrate its utility and generalizability, two different datasets were explored using ARGUS and our use cases and designs were strongly validated in evaluation sessions with expert and non-expert users.

In big data applications from digital health to assisted living smart systems, only a fraction of data instances used for training classifiers t end to be labeled. One important subfield of weakly labeled learning, called Positive Unlabeled (PU) learning, does not require a completely labeled dataset in order to train a strong classifier. This is crucial as in many domains it is expensive or impossible to obtain a completely labeled dataset. While prior PU work assumed that unlabeled instances occurred with a random uniform distribution, we observe that labeled (and unlabeled) data tends to occur in long contiguous sequences (or bursts) due the prevalent burst labeling behavior by human annotators. Burst labeling leads to a sequential bias in PU data not addressed by state-of-the-art methods. To tackle this open problem of learning under sequential bias, we propose BurstPU, the first framework for training a classifier on sequentially labeled PU data. BurstPU addresses the challenge that two interdependent models must be learned, namely, the classification model and the labeling likelihood model, with the later predicting the likelihood that a given instance is labeled. The labeling likelihood model is then needed during the training of the classification model to account for the bias in the labeling process. Our experimental study demonstrates that BurstPU consistently outperforms all state-of-the-art PU methods on a rich variety of diverse real-world datasets, and can learn from fewer labeled instances compared to state-of-art PU methods.

Current research in passive health monitoring utilizes machine learning methods to infer users' symptoms and health status from smartphone-sensed data, which can be gathered on a large scale. However, reasoning about smartphone-sensed health behaviors should engage health experts who may not want to solely rely on computational approaches as they provide limited insights. We designed and proposed a visualization framework for the INTeractive Observation of Smartphone-Inferred Symptoms (INTOSIS), that supports contextualization of symptomatic days by presenting a holistic picture of complex smartphone data for analysts to find c oncerning b ehavior patterns. For instance, while sedentary behavior caused by the flu is concerning, sedentary behaviors on holidays are non-concerning. INTOSIS visualizes multiple smartphone sensor data channels such as geo-location, app usage and screen usage. It uses visual metaphors to effectively represent the data to help analysts derive important human-understandable spatio-temporal contexts and assign health consequences with corresponding semantic labels. INTOSIS provides timeline visualizations of contextual clues such as screen and app usage at night that support an analyst in being able to reason about and then extract plausible explanations for the occurrence of certain symptoms, such as, sleep problems. We validate INTOSIS with intuitive use cases, using a real-world smartphone-sensed dataset, along with expert evaluation.

Context-Aware (CA) systems that adapt their behavior based on their users' current context have broad applications in areas from healthcare to smart environments. Context Recognition (CR) is currently solved using machine and deep learning approaches that require realistic datasets with ground truth labels collected "in-the-wild" as users live their lives. In such studies, users periodically label their current context (current activity, body position, and location) while a mobile app continuously gathers sensor data. Unfortunately, users sometimes assign wrong or incomplete context labels, reducing the quality of the labeled dataset; and causing lower classification accuracy. We present COMEX, an interactive visual analytics tool that assists analysts in identifying instances of mislabeled context data to improve the quality of CA datasets. For this, we first provide a conceptual categorization of mislabeling error types. Thereafter we develop linked visualizations, augmented by anomaly scores indicating suspected labeling issues, which provide richer insights into the diverse characteristics of the target dataset. We validate our approach on an open source dataset that contains context information for 60 participants gathered over several days using smartphones. With the help of COMEX, participants of our case study identified numerous mislabelled instances in the dataset. We re-ran the classification task after excluding mislabelled data and saw improvements in classification accuracy.