MoodCapture presents a novel approach that assesses depression based on images automatically captured from the front-facing camera of smartphones as people go about their daily lives. We collect over 125,000 photos in the wild from N=177 participants diagnosed with major depressive disorder for 90 days. Images are captured naturalistically while participants respond to the PHQ-8 depression survey question: “I have felt down, depressed, or hopeless”. Our analysis explores important image attributes, such as angle, dominant colors, location, objects, and lighting. We show that a random forest trained with face landmarks can classify samples as depressed or non-depressed and predict raw PHQ-8 scores effectively. Our post-hoc analysis provides several insights through an ablation study, feature importance analysis, and bias assessment. Importantly, we evaluate user concerns about using MoodCapture to detect depression based on sharing photos, providing critical insights into privacy concerns that inform the future design of in-the-wild image-based mental health assessment tools.

The COVID-19 pandemic continues to affect the daily life of college students, impacting their social life, education, stress levels and overall mental well-being. We study and assess behavioral changes of N=180 undergraduate college students one year prior to the pandemic as a baseline and then during the first year of the pandemic using mobile phone sensing and behavioral inference. We observe that certain groups of students experience the pandemic very differently. Furthermore, we explore the association of self-reported COVID-19 concern with students' behavior and mental health. We find that heightened COVID-19 concern is correlated with increased depression, anxiety and stress. We evaluate the performance of different deep learning models to classify student COVID-19 concerns with an AUROC and F1 score of 0.70 and 0.71, respectively. Our study spans a two-year period and provides a number of important insights into the life of college students during this period.

Pandemics significantly impact human daily life. People throughout the world adhere to safety protocols (e.g., social distancing and self-quarantining). As a result, they willingly keep distance from workplace, friends and even family. In such circumstances, in-person social interactions may be substituted with virtual ones via online channels, such as, Instagram and Snapchat. To get insights into this phenomenon, we study a group of undergraduate students before and after the start of COVID-19 pandemic. Specifically, we track N=102 undergraduate students on a small college campus prior to the pandemic using mobile sensing from phones and assign semantic labels to each location they visit on campus where they study, socialize and live. By leveraging their colocation network at these various semantically labeled places on campus, we find that colocations at certain places that possibly proxy higher in-person social interactions (e.g., dormitories, gyms and Greek houses) show significant predictive capability in identifying the individuals' change in social media usage during the pandemic period. We show that we can predict student's change in social media usage during COVID-19 with an F1 score of 0.73 purely from the in-person colocation data generated prior to the pandemic.

Impaired social functioning is a symptom of mental illness (e.g., depression, schizophrenia) and a wide range of other conditions (e.g., cognitive decline in the elderly, dementia). Today, assessing social functioning relies on subjective evaluations and self assessments. We propose a different approach and collect detailed social functioning measures and objective mobile sensing data from N=55 outpatients living with schizophrenia to study new methods of passively accessing social functioning. We identify a number of behavioral patterns from sensing data, and discuss important correlations between social function sub-scales and mobile sensing features. We show we can accurately predict the social functioning of outpatients in our study including the following sub-scales: prosocial activities (MAE = 7.79, r = 0.53), which indicates engagement in common social activities; interpersonal behavior (MAE = 3.39, r = 0.57), which represents the number of friends and quality of communications; and employment/occupation (MAE = 2.17, r = 0.62), which relates to engagement in productive employment or a structured program of daily activity. Our work on automatically inferring social functioning opens the way to new forms of assessment and intervention across a number of areas including mental health and aging in place.

Schizophrenia is a severe psychiatric disorder. We use the CrossCheck study dataset to develop methods to predict whether or not a patient with schizophrenia is going to relapse from mobile phone data. Out of 75 patients in the year long randomized controlled trial only 27 relapse episodes occur. We apply various techniques to address predicting rare events in a longitudinal dataset. We apply resampling methods combining oversampling relapse examples and undersampling non-relapse examples and impute missing data. To avoid overfitting, we apply feature selection and transformation (i.e., PCA) to reduce the feature dimensionality. We find the best relapse prediction result using the first 100 principal components from both passive sensing and self-reports with 30-day prediction windows (precision=26.8%, recall=28.4%). If we demand the recall to be greater than 50%, we find the best result using 25 principle components from both passive sensing and self-reports with 30-day prediction windows (precision=15.4%, recall=51.6%).