MoodCapture presents a novel approach that assesses depression based on images automatically captured from the front-facing camera of smartphones as people go about their daily lives. We collect over 125,000 photos in the wild from N=177 participants diagnosed with major depressive disorder for 90 days. Images are captured naturalistically while participants respond to the PHQ-8 depression survey question: “I have felt down, depressed, or hopeless”. Our analysis explores important image attributes, such as angle, dominant colors, location, objects, and lighting. We show that a random forest trained with face landmarks can classify samples as depressed or non-depressed and predict raw PHQ-8 scores effectively. Our post-hoc analysis provides several insights through an ablation study, feature importance analysis, and bias assessment. Importantly, we evaluate user concerns about using MoodCapture to detect depression based on sharing photos, providing critical insights into privacy concerns that inform the future design of in-the-wild image-based mental health assessment tools.

Impaired social functioning is a symptom of mental illness (e.g., depression, schizophrenia) and a wide range of other conditions (e.g., cognitive decline in the elderly, dementia). Today, assessing social functioning relies on subjective evaluations and self assessments. We propose a different approach and collect detailed social functioning measures and objective mobile sensing data from N=55 outpatients living with schizophrenia to study new methods of passively accessing social functioning. We identify a number of behavioral patterns from sensing data, and discuss important correlations between social function sub-scales and mobile sensing features. We show we can accurately predict the social functioning of outpatients in our study including the following sub-scales: prosocial activities (MAE = 7.79, r = 0.53), which indicates engagement in common social activities; interpersonal behavior (MAE = 3.39, r = 0.57), which represents the number of friends and quality of communications; and employment/occupation (MAE = 2.17, r = 0.62), which relates to engagement in productive employment or a structured program of daily activity. Our work on automatically inferring social functioning opens the way to new forms of assessment and intervention across a number of areas including mental health and aging in place.

The analysis of wearable-sensory time series data (e.g., heart rate records) benefits many applications (e.g., activity recognition, disease diagnosis). However, sensor measurements usually contain missing values due to various factors (e.g., user behavior, lack of charging), which may degrade the performance of downstream analytical tasks (e.g., regression, prediction). Thus, time series imputation is desired, which is capable of making sensory time series complete. Existing time series imputation methods generally employ various deep neural network models (e.g., GRU and GAN) to fill missing values by leveraging temporal patterns extracted from the contextual observations. Despite their effectiveness, we argue that most existing models can only achieve sub-optimal imputation performance due to the fact that they are inherently limited in sharing only one single set of model parameters to perform imputation on all individuals. Relying on one set of parameters limits the expressiveness of the imputation model as such models are bound to fail in capturing various complex personal characteristics. Therefore, most existing models tend to achieve inferior imputation performance, especially when a long duration of missing values, i.e., a large gap, is observed in the time series data. To address the limitation, this work develops a new imputation framework--Personalized Wearable-Sensory Time Series Imputation framework (PTSI) to provide a fully personalized treatment for time series imputation via effective knowledge transfer. In particular, PTSI first leverages a meta-learning paradigm to learn a well-generalized initialization to facilitate the adaption process for each user. To make the time series imputation be reflective of an individual's unique characteristics, we further endow PTSI with the capability of learning personalized model parameters, which is achieved by designing a parameter initialization modulating component. Extensive experiments on real-world human heart rate datasets demonstrate that our PTSI framework outperforms various state-of-the-art methods by a large margin consistently.

The advancement of artificial intelligence and wearable computing triggers the radical innovation of cognitive applications. In this work, we propose VIMES, an augmented reality-based memory assistance system that helps recall declarative memory, such as whom the user meets and what they chat. Through a collaborative method with 20 participants, we design VIMES, a system that runs on smartglasses, takes the first-person audio and video as input, and extracts personal profiles and event information to display on the embedded display or a smartphone. We perform an extensive evaluation with 50 participants to show the effectiveness of VIMES for memory recall. VIMES outperforms (90% memory accuracy) other traditional methods such as self-recall (34%) while offering the best memory experience (Vividness, Coherence, and Visual Perspective all score over 4/5). The user study results show that most participants find VIMES useful (3.75/5) and easy to use (3.46/5).

The ubiquitous use of social media enables researchers to obtain self-recorded longitudinal data of individuals in real-time. Because this data can be collected in an inexpensive and unobtrusive way at scale, social media has been adopted as a “passive sensor” to study human behavior. However, such research is impacted by the lack of homogeneity in the use of social media, and the engineering challenges in obtaining such data. This paper proposes a statistical framework to leverage the potential of social media in sensing studies of human behavior, while navigating the challenges associated with its sparsity. Our framework is situated in a large-scale in-situ study concerning the passive assessment of psychological constructs of 757 information workers wherein of four sensing streams was deployed - bluetooth beacons, wearable, smartphone, and social media. Our framework includes principled feature transformation and machine learning models that predict latent social media features from the other passive sensors. We demonstrate the efficacy of this imputation framework via a high correlation of 0.78 between actual and imputed social media features. With the imputed features we test and validate predictions on psychological constructs like personality traits and affect. We find that adding the social media data streams, in their imputed form, improves the prediction of these measures. We discuss how our framework can be valuable in multimodal sensing studies that aim to gather comprehensive signals about an individual's state or situation.