Health information technologies are transforming how mental healthcare is paid for through value-based care programs, which tie payment to data quantifying care outcomes. But, it is unclear what outcomes data these technologies should store, how to engage users in data collection, and how outcomes data can improve care. Given these challenges, we conducted interviews with 30 U.S.-based mental health clinicians to explore the design space of health information technologies that support outcomes data specification, collection, and use in value-based mental healthcare. Our findings center clinicians' perspectives on aligning outcomes data for payment programs and care; opportunities for health technologies and personal devices to improve data collection; and considerations for using outcomes data to hold stakeholders including clinicians, health insurers, and social services financially accountable in value-based mental healthcare. We conclude with implications for future research designing and developing technologies supporting value-based care across stakeholders involved with mental health service delivery.

Despite the prevalence of sleep-tracking devices, many individuals struggle to translate data into actionable improvements in sleep health. Current methods often provide data-driven suggestions but may not be feasible and adaptive to real-life constraints and individual contexts. We present HealthGuru, a novel large language model-powered chatbot to enhance sleep health through data-driven, theory-guided, and adaptive recommendations with conversational behavior change support. HealthGuru's multi-agent framework integrates wearable device data, contextual information, and a contextual multi-armed bandit model to suggest tailored sleep-enhancing activities. The system facilitates natural conversations while incorporating data-driven insights and theoretical behavior change techniques. Our eight-week in-the-wild deployment study with 16 participants compared HealthGuru to a baseline chatbot. Results show improved metrics like sleep duration and activity scores, higher quality responses, and increased user motivation for behavior change with HealthGuru. We also identify challenges and design considerations for personalization and user engagement in health chatbots.

Recent research has explored computational tools to manage workplace stress via personal sensing, a measurement paradigm in which behavioral data streams are collected from technologies including smartphones, wearables, and personal computers. As these tools develop, they invite inquiry into how they can be appropriately implemented towards improving workers' well-being. In this study, we explored this proposition through formative interviews followed by a design provocation centered around measuring burnout in a U.S. resident physician program. Residents and their supervising attending physicians were presented with medium-fidelity mockups of a dashboard providing behavioral data on residents' sleep, activity and time working; self-reported data on residents' levels of burnout; and a free text box where residents could further contextualize their well-being. Our findings uncover tensions around how best to measure workplace well-being, who within a workplace is accountable for worker stress, and how the introduction of such tools remakes the boundaries of appropriate information flows between worker and workplace. We conclude by charting future work confronting these tensions, to ensure personal sensing is leveraged to truly improve worker well-being.

The proliferation of e-cigarettes and portable vaporizers presents new opportunities for accurately and unobtrusively tracking e-cigarette use. PuffPacket is a hardware and soft-ware research platform that leverages the technology built into vaporizers, e-cigarettes and other electronic drug delivery devices to ubiquitously track their usage. The system piggybacks on the signals these devices use to directly measure and track the nicotine consumed by users. PuffPacket augments e-cigarettes with Bluetooth to calculate the frequency, intensity, and duration of each inhalation. This information is augmented with smartphone-based location and activity information to help identify potential contextual triggers. Puff-Packet is generalizable to a wide variety of electronic nicotine,THC, and other drug delivery devices currently on the mar-ket. The hardware and software for PuffPacket is open-source so it can be expanded upon and leveraged for mobile health tracking research.

Impaired social functioning is a symptom of mental illness (e.g., depression, schizophrenia) and a wide range of other conditions (e.g., cognitive decline in the elderly, dementia). Today, assessing social functioning relies on subjective evaluations and self assessments. We propose a different approach and collect detailed social functioning measures and objective mobile sensing data from N=55 outpatients living with schizophrenia to study new methods of passively accessing social functioning. We identify a number of behavioral patterns from sensing data, and discuss important correlations between social function sub-scales and mobile sensing features. We show we can accurately predict the social functioning of outpatients in our study including the following sub-scales: prosocial activities (MAE = 7.79, r = 0.53), which indicates engagement in common social activities; interpersonal behavior (MAE = 3.39, r = 0.57), which represents the number of friends and quality of communications; and employment/occupation (MAE = 2.17, r = 0.62), which relates to engagement in productive employment or a structured program of daily activity. Our work on automatically inferring social functioning opens the way to new forms of assessment and intervention across a number of areas including mental health and aging in place.

Schizophrenia is a severe psychiatric disorder. We use the CrossCheck study dataset to develop methods to predict whether or not a patient with schizophrenia is going to relapse from mobile phone data. Out of 75 patients in the year long randomized controlled trial only 27 relapse episodes occur. We apply various techniques to address predicting rare events in a longitudinal dataset. We apply resampling methods combining oversampling relapse examples and undersampling non-relapse examples and impute missing data. To avoid overfitting, we apply feature selection and transformation (i.e., PCA) to reduce the feature dimensionality. We find the best relapse prediction result using the first 100 principal components from both passive sensing and self-reports with 30-day prediction windows (precision=26.8%, recall=28.4%). If we demand the recall to be greater than 50%, we find the best result using 25 principle components from both passive sensing and self-reports with 30-day prediction windows (precision=15.4%, recall=51.6%).

Motivated by the need to support those self-managing chronic pain, we report on the development and evaluation of a novel pressure-based tangible user interface (TUI) for the self-report of scalar values representing pain intensity. Our TUI consists of a conductive foam-based, force-sensitive resistor (FSR) covered in a soft rubber with embedded signal conditioning, an ARM Cortex-M0 microprocessor, and Bluetooth Low Energy (BLE). In-lab usability and feasibility studies with 28 participants found that individuals were able to use the device to make reliable reports with four degrees of freedom as well map squeeze pressure to pain level and visual feedback. Building on insights from these studies, we further redesigned the FSR into a wearable device with multiple form factors, including a necklace, bracelet, and keychain. A usability study with an additional 7 participants from our target population, elderly individuals with chronic pain, found high receptivity to the wearable design, which offered a number of participant-valued characteristics (e.g., discreetness) along with other design implications that serve to inform the continued refinement of tangible devices that support pain self-assessment.

Emotions play a major role in how interpersonal conflicts unfold. Although several strategies and technological approaches have been proposed for emotion regulation, they often require conscious attention and effort. This often limits their efficacy in practice. In this paper, we propose a different approach inspired by self-perception theory: noticing that people are often reacting to the perception of their own behavior, we artificially change their perceptions to influence their emotions. We conducted two studies to evaluate the potential of this approach by automatically and subtly altering how people perceive their own voice. In one study, participants that received voice feedback with a calmer tone during relationship conflicts felt less anxious. In the other study, participants who listened to their own voices with a lower pitch during contentious debates felt more powerful. We discuss the implications of our findings and the opportunities for designing automatic and less perceptible emotion regulation systems.

Alertness is a crucial component of our cognitive performance. Reduced alertness can negatively impact memory consolidation, productivity and safety. As a result, there has been an increasing focus on continuous assessment of alertness. The existing methods usually require users to wear sensors, fill out questionnaires, or perform response time tests periodically, in order to track their alertness. These methods may be obtrusvie to some users, and thus have limited capability. In this work, we propose AlertnessScanner, a computer-vision-based system that collects in-situ pupil information to model alertness in the wild. We conducted two in-the-wild studies to evaluate the effectiveness of our solution, and found that AlertnessScanner passively and unobtrusively assess alertness. We discuss the implications of our findings and present opportunities for mobile applications that measure and act upon changes in alertness.

Early detection of mental health changes in individuals with serious mental illness is critical for effective intervention. CrossCheck is the first step towards the passive monitoring of mental health indicators in patients with schizophrenia and paves the way towards relapse prediction and early intervention. In this paper, we present initial results from an ongoing randomized control trial, where passive smartphone sensor data is collected from 21 outpatients with schizophrenia recently discharged from hospital over a period ranging from 2-8.5 months. Our results indicate that there are statistically significant associations between automatically tracked behavioral features related to sleep, mobility, conversations, smart-phone usage and self-reported indicators of mental health in schizophrenia. Using these features we build inference models capable of accurately predicting aggregated scores of mental health indicators in schizophrenia with a mean error of 7.6% of the score range. Finally, we discuss results on the level of personalization that is needed to account for the known variations within people. We show that by leveraging knowledge from a population with schizophrenia, it is possible to train accurate personalized models that require fewer individual-specific data to quickly adapt to new users.

Throughout the day, our alertness levels change and our cognitive performance fluctuates. The creation of technology that can adapt to such variations requires reliable measurement with ecological validity. Our study is the first to collect alertness data in the wild using the clinically validated Psychomotor Vigilance Test. With 20 participants over 40 days, we find that alertness can oscillate approximately 30% depending on time and body clock type and that Daylight Savings Time, hours slept, and stimulant intake can influence alertness as well. Based on these findings, we develop novel methods for unobtrusively and continuously assessing alertness. In estimating response time, our model achieves a root-mean-square error of 80.64 milliseconds, which is significantly lower than the 500ms threshold used as a standard indicator of impaired cognitive ability. Finally, we discuss how such real-time detection of alertness is a key first step towards developing systems that are sensitive to our biological variations.

In this paper we demonstrate that it is possible to help individuals regulate their emotions with mobile interventions that leverage the way we naturally react to our bodily signals. Previous studies demonstrate that the awareness of our bodily signals, such as our heart rate, directly influences the way we feel. By leveraging these findings we designed a wearable device to regulate user's anxiety by providing a false feedback of a slow heart rate. The results of an experiment with 67 participants show that the device kept the anxiety of the individuals in low levels when compared to the control group and the other conditions. We discuss the implications of our findings and present some promising directions for designing and developing this type of intervention for emotion regulation.

Our body clock causes considerable variations in our behavioral, mental, and physical processes, including alertness, throughout the day. While much research has studied technology usage patterns, the potential impact of underlying biological processes on these patterns is under-explored. Using data from 20 participants over 40 days, this paper presents the first study to connect patterns of mobile application usage with these contributing biological factors. Among other results, we find that usage patterns vary for individuals with different body clock types, that usage correlates with rhythms of alertness, that app use features such as duration and switching can distinguish periods of low and high alertness, and that app use reflects sleep interruptions as well as sleep duration. We conclude by discussing how our findings inform the design of biologically-friendly technology that can better support personal rhythms of performance.

In this paper, we propose Nutrilyzer, a novel mobile sensing system for characterizing the nutrients and detecting adulterants in liquid food with the photoacoustic effect. By listening to the sound of the intensity modulated light or electromagnetic wave with different wavelengths, our mobile photoacoustic sensing system captures unique spectra produced by the transmitted and scattered light while passing through various liquid food. As different liquid foods with different chemical compositions yield uniquely different spectral signatures, Nutrilyzer's signal processing and machine learning algorithm learn to map the photoacoustic signature to various liquid food characteristics including nutrients and adulterants. We evaluated Nutrilyzer for milk nutrient prediction (i.e., milk protein) and milk adulterant detection. We have also explored Nutrilyzer for alcohol concentration prediction. The Nutrilyzer mobile system consists of an array of 16 LEDs in ultraviolet, visible and near-infrared region, two piezoelectric sensors and an ARM microcontroller unit, which are designed and fabricated in a printed circuit board and a 3D printed photoacoustic housing.

OBJECTIVE: To evaluate the feasibility of automatically assessing the Social Rhythm Metric (SRM), a clinically-validated marker of stability and rhythmicity for individuals with bipolar disorder (BD), using passively-sensed data from smartphones. METHODS: Seven patients with BD used smartphones for 4 weeks passively collecting sensor data including accelerometer, microphone, location, and communication information to infer behavioral and contextual patterns. Participants also completed SRM entries using a smartphone app. RESULTS: We found that automated sensing can be used to infer the SRM score. Using location, distance traveled, conversation frequency, and non-stationary duration as inputs, our generalized model achieves root-mean-square-error of 1.40, a reasonable performance given the range of SRM score (0-7). Personalized models further improve performance with mean root-mean-square-error of 0.92 across users. Classifiers using sensor streams can predict stable (SRM score ≥3.5) and unstable (SRM score <3.5) states with high accuracy (precision: 0.85 and recall: 0.86). CONCLUSIONS: Automatic smartphone sensing is a feasible approach for inferring rhythmicity, a key marker of wellbeing for individuals with BD.

The increasing adoption of social media provides unprecedented opportunities to gain insight into human nature at vastly broader scales. Regarding the study of population-wide sentiment, prior research commonly focuses on text-based analyses and ignores a treasure trove of sentiment-laden content: images. In this paper, we make methodological and computational contributions by introducing the Smile Index as a formalized measure of societal happiness. Detecting smiles in 9 million geo-located tweets over 16 months, we validate our Smile Index against both text-based techniques and self-reported happiness. We further make observational contributions by applying our metric to explore temporal trends in sentiment, relate public mood to societal events, and predict economic indicators. Reflecting upon the innate, language-independent aspects of facial expressions, we recommend future improvements and applications to enable robust, global-level analyses. We conclude with implications for researchers studying and facilitating the expression of collective emotion through socio-technical systems.

Persuasive technologies aim to influence user's behaviors. In order to be effective, many of the persuasive technologies de-veloped so far relies on user's motivation and ability, which is highly variable and often the reason behind the failure of such technology. In this paper, we present the concept of Mindless Computing, which is a new approach to persuasive technology design. Mindless Computing leverages theories and concepts from psychology and behavioral economics into the design of technologies for behavior change. We show through a systematic review that most of the current persuasive technologies do not utilize the fast and automatic mental processes for behavioral change and there is an opportunity for persuasive technology designers to develop systems that are less reliant on user's motivation and ability. We describe two examples of mindless technologies and present pilot studies with encouraging results. Finally, we discuss design guidelines and considerations for developing this type of persuasive technology.

By nature, we are circadian creatures whose bodies' biological clocks drive numerous physiological, mental, and behavioral rhythms. Simultaneously, we are social beings. Accordingly, our internal circadian timings experience interference from externally determined factors such as work schedules and social engagements, and digital connectivity imports additional social constraints that can further misalign our individual body clocks. Misalignment between biological and social time causes social jet lag [50], which has serious physical and mental health consequences. It particularly impacts our sleep processes and neurobehavioral functioning. Examining the interplay between biological rhythms and technology-mediated social interactions, we find that technology may both modulate and reflect circadian rhythms. We also leverage such social-sensor data to infer sleep-related behaviors and disruptions and to analyze variations in attention, cognitive performance, and mood following (in)adequate sleep. We conclude with recommendations for designing technologies attuned to our innate biological traits.

Mobile sensing systems have made significant advances in tracking human behavior. However, the development of personalized mobile health feedback systems is still in its infancy. This paper introduces MyBehavior, a smartphone application that takes a novel approach to generate deeply personalized health feedback. It combines state-of-the-art behavior tracking with algorithms that are used in recommendation systems. MyBehavior automatically learns a user's physical activity and dietary behavior and strategically suggests changes to those behaviors for a healthier lifestyle. The system uses a sequential decision making algorithm, Multi-armed Bandit, to generate suggestions that maximize calorie loss and are easy for the user to adopt. In addition, the system takes into account user's preferences to encourage adoption using the pareto-frontier algorithm. In a 14-week study, results show statistically significant increases in physical activity and decreases in food calorie when using MyBehavior compared to a control condition.

In this paper, we present DoppleSleep -- a contactless sleep sensing system that continuously and unobtrusively tracks sleep quality using commercial off-the-shelf radar modules. DoppleSleep provides a single sensor solution to track sleep-related physical and physiological variables including coarse body movements and subtle and fine-grained chest, heart movements due to breathing and heartbeat. By integrating vital signals and body movement sensing, DoppleSleep achieves 89.6% recall with Sleep vs. Wake classification and 80.2% recall with REM vs. Non-REM classification compared to EEG-based sleep sensing. Lastly, it provides several objective sleep quality measurements including sleep onset latency, number of awakenings, and sleep efficiency. The contactless nature of DoppleSleep obviates the need to instrument the user's body with sensors. Lastly, DoppleSleep is implemented on an ARM microcontroller and a smartphone application that are benchmarked in terms of power and resource usage.

In this paper, we argue that atypical cognitive, perceptual and behavioral characteristics associated with serious mental illnesses should be taken into consideration when designing health technologies. While applications have been developed to assist in the treatment of these illnesses, the specific psychological characteristics of these disorders have rarely been considered extensively in the design process. Here, we explore how an understanding of the low-level characteristics of bipolar disorder, combined with a clinically-validated treatment and patients' lived experience, can inform mHealth design. We present a novel method -- in situ design -- to support ecologically valid design, and demonstrate its use through the co-development with 9 individuals with bipolar disorder of MoodRhythm, a mobile application designed to track and stabilize daily routines. We provide evidence that mHealth design elements tailored to the characteristics and needs of individuals with bipolar disorder can result in engaging interactions.

We often think of ourselves as individuals with steady capabilities. However, converging strands of research indicate that this is not the case. Our biochemistry varies significantly over the course of a 24 hour period. Consequently our levels of alertness, productivity, physical activity, and even sensitivity to pain fluctuate throughout the day. This offers a considerable opportunity for the UbiComp community to identify novel measurements and interventions that can leverage these daily variations. To illustrate this potential, we present results from an empirical study with 9 participants over 97 days investigating whether such variations manifest in low-level smartphone use, focusing on daily rhythms related to sleep. Our findings demonstrate that phone usage patterns can be used to detect and predict individual daily variations indicative of temporal preference, sleep duration, and deprivation. We also identify opportunities and challenges for measuring and enhancing well-being using these simple and effective markers of circadian rhythms.

In this paper, we propose BodyBeat, a novel mobile sensing system for capturing and recognizing a diverse range of non-speech body sounds in real-life scenarios. Non-speech body sounds, such as sounds of food intake, breath, laughter, and cough contain invaluable information about our dietary behavior, respiratory physiology, and affect. The BodyBeat mobile sensing system consists of a custom-built piezoelectric microphone and a distributed computational framework that utilizes an ARM microcontroller and an Android smartphone. The custom-built microphone is designed to capture subtle body vibrations directly from the body surface without being perturbed by external sounds. The microphone is attached to a 3D printed neckpiece with a suspension mechanism. The ARM embedded system and the Android smartphone process the acoustic signal from the microphone and identify non-speech body sounds. We have extensively evaluated the BodyBeat mobile sensing system. Our results show that BodyBeat outperforms other existing solutions in capturing and recognizing different types of important non-speech body sounds.

Visual approaches for conducting research during the design process often give voice to people and ideas that might otherwise remain obscured. Recent and increasing interest in visual research techniques has coincided with technological advances such as camera phones and visually oriented mobile applications. As a result of this close association between digital technologies and image-based research techniques, there are multiple opportunities and challenges within human–computer interaction (HCI) design practice to employ these strategies to improve user experiences. This article provides an overview of current visual approaches to research highlighting the role technology has played in facilitating and inspiring these techniques. A series of case studies are presented that provide a basis for understanding a breadth of visual approaches in HCI design practices as well as serve as a point of entry to a critical and reflective discussion about the use of these approaches in different circumstances. Based on these reflections, three value statements are offered as a means to encourage the use of these visual approaches more broadly and critically in HCI design studies.

The rising popularity of the sensor-equipped smartphone is changing the possible scale and scope of human activity inference. The diversity in user population seen in large user bases can overwhelm conventional one-size-fits-all classification approaches. Although personalized models are better able to handle population diversity, they often require increased effort from the end user during training and are computationally expensive. In this paper, we propose an activity classification framework that is scalable and can tractably handle an increasing number of users. Scalability is achieved by maintaining distinct groups of similar users during the training process, which makes it possible to account for the differences between users without resorting to training individualized classifiers. The proposed framework keeps user burden low by leveraging crowd-sourced data labels, where simple natural language processing techniques in combination with multi-instance learning are used to handle labeling errors introduced by low-commitment everyday users. Experiment results on a large public dataset demonstrate that the framework can cope with population diversity irrespective of population size.

Stress can have long term adverse effects on individuals' physical and mental well-being. Changes in the speech production process is one of many physiological changes that happen during stress. Microphones, embedded in mobile phones and carried ubiquitously by people, provide the opportunity to continuously and non-invasively monitor stress in real-life situations. We propose StressSense for unobtrusively recognizing stress from human voice using smartphones. We investigate methods for adapting a one-size-fits-all stress model to individual speakers and scenarios. We demonstrate that the StressSense classifier can robustly identify stress across multiple individuals in diverse acoustic environments: using model adaptation StressSense achieves 81% and 76% accuracy for indoor and outdoor environments, respectively. We show that StressSense can be implemented on commodity Android phones and run in real-time. To the best of our knowledge, StressSense represents the first system to consider voice based stress detection and model adaptation in diverse real-life conversational situations using smartphones.

The emerging field of mobile sensing has engaged computer scientists from a variety of existing communities, such as, mobile systems, machine learning and human computer interaction. Each community approaches the challenges of mobile sensing research with its own unique perspective. The purpose of this workshop is to provide a forum to discuss the state of the art in mobile sensing and promote increased cooperation and interaction among the participating research communities.

Sensor-enabled smartphones are opening a new frontier in the development of mobile sensing applications. The recognition of human activities and context from sensor-data using classification models underpins these emerging applications. However, conventional approaches to training classifiers struggle to cope with the diverse user populations routinely found in large-scale popular mobile applications. Differences between users (e.g., age, sex, behavioral patterns, lifestyle) confuse classifiers, which assume everyone is the same. To address this, we propose Community Similarity Networks (CSN), which incorporates inter-person similarity measurements into the classifier training process. Under CSN every user has a unique classifier that is tuned to their own characteristics. CSN exploits crowd-sourced sensor-data to personalize classifiers with data contributed from other similar users. This process is guided by similarity networks that measure different dimensions of inter-person similarity. Our experiments show CSN outperforms existing approaches to classifier training under the presence of population diversity.

The idea of continuously monitoring well-being using mobile-sensing systems is gaining popularity. In-situ measurement of human behavior has the potential to overcome the short comings of gold-standard surveys that have been used for decades by the medical community. However, current sensing systems have mainly focused on tracking physical health; some have approximated aspects of mental health based on proximity measurements but have not been compared against medically accepted screening instruments. In this paper, we show the feasibility of a multi-modal mobile sensing system to simultaneously assess mental and physical health. By continuously capturing fine grained motion and privacy-sensitive audio data, we are able to derive different metrics that reflect the results of commonly used surveys for assessing well-being by the medical community. In addition, we present a case study that highlights how errors in assessment due to the subjective nature of the responses could potentially be avoided by continuous sensing and inference of social interactions and physical activities.

New technologies have made it possible to collect information about social networks as they are acted and observed in the wild , instead of as they are reported in retrospective surveys. These technologies offer opportunities to address many new research questions: How can meaningful information about social interaction be extracted from automatically recorded raw data on human behavior? What can we learn about social networks from such fine-grained behavioral data? And how can all of this be done while protecting privacy? With the goal of addressing these questions, this article presents new methods for inferring colocation and conversation networks from privacy-sensitive audio. These methods are applied in a study of face-to-face interactions among 24 students in a graduate school cohort during an academic year. The resulting analysis shows that networks derived from colocation and conversation inferences are quite different. This distinction can inform future research in computational social science, especially work that only measures colocation or employs colocation data as a proxy for conversation networks.

Modeling human behavior requires vast quantities of accurately labeled training data, but for ubiquitous people-aware applications such data is rarely attainable. Even researchers make mistakes when labeling data, and consistent, reliable labels from low-commitment users are rare. In particular, users may give identical labels to activities with characteristically different signatures (e.g., labeling eating at home or at a restaurant as "dinner") or may give different labels to the same context (e.g., "work" vs. "office"). In this scenario, labels are unreliable but nonetheless contain valuable information for classification. To facilitate learning in such unconstrained labeling scenarios, we propose Community-Guided Learning (CGL), a framework that allows existing classifiers to learn robustly from unreliably-labeled user-submitted data. CGL exploits the underlying structure in the data and the unconstrained labels to intelligently group crowd-sourced data. We demonstrate how to use similarity measures to determine when and how to split and merge contributions from different labeled categories and present experimental results that demonstrate the effectiveness of our framework.

The current methods used to mine and analyze temporal social network data make two assumptions: all edges have the same strength, and all parameters are time-homogeneous. We show that those assumptions may not hold for social networks and propose an alternative model with two novel aspects: (1) the modeling of edges as multi-valued variables that can change in intensity, and (2) the use of a curved exponential family framework to capture time-inhomogeneous properties while retaining a parsimonious and interpretable model. We show that our model outperforms traditional models on two real-world social network data sets.

We present Darwin, an enabling technology for mobile phone sensing that combines collaborative sensing and classification techniques to reason about human behavior and context on mobile phones. Darwin advances mobile phone sensing through the deployment of efficient but sophisticated machine learning techniques specifically designed to run directly on sensor-enabled mobile phones (i.e., smartphones). Darwin tackles three key sensing and inference challenges that are barriers to mass-scale adoption of mobile phone sensing applications: (i) the human-burden of training classifiers, (ii) the ability to perform reliably in different environments (e.g., indoor, outdoor) and (iii) the ability to scale to a large number of phones without jeopardizing the "phone experience" (e.g., usability and battery lifetime). Darwin is a collaborative reasoning framework built on three concepts: classifier/model evolution, model pooling, and collaborative inference. To the best of our knowledge Darwin is the first system that applies distributed machine learning techniques and collaborative inference concepts to mobile phones. We implement the Darwin system on the Nokia N97 and Apple iPhone. While Darwin represents a general framework applicable to a wide variety of emerging mobile sensing applications, we implement a speaker recognition application and an augmented reality application to evaluate the benefits of Darwin. We show experimental results from eight individuals carrying Nokia N97s and demonstrate that Darwin improves the reliability and scalability of the proof-of-concept speaker recognition application without additional burden to users.

Supporting continuous sensing applications on mobile phones is challenging because of the resource demands of long-term sensing, inference and communication algorithms. We present the design, implementation and evaluation of the Jigsaw continuous sensing engine, which balances the performance needs of the application and the resource demands of continuous sensing on the phone. Jigsaw comprises a set of sensing pipelines for the accelerometer, microphone and GPS sensors, which are built in a plug and play manner to support: i) resilient accelerometer data processing, which allows inferences to be robust to different phone hardware, orientation and body positions; ii) smart admission control and on-demand processing for the microphone and accelerometer data, which adaptively throttles the depth and sophistication of sensing pipelines when the input data is low quality or uninformative; and iii) adaptive pipeline processing, which judiciously triggers power hungry pipeline stages (e.g., sampling the GPS) taking into account the mobility and behavioral patterns of the user to drive down energy costs. We implement and evaluate Jigsaw on the Nokia N95 and the Apple iPhone, two popular smartphone platforms, to demonstrate its capability to recognize user activities and perform long term GPS tracking in an energy-efficient manner.

Mobile medical sensors promise to provide an efficient, accurate, and economic way to monitor patients' health outside the hospital. Patient authentication is a necessary security requirement in remote health monitoring scenarios. The monitoring system needs to make sure that the data is coming from the right person before any medical or financial decisions are made based on the data. Credential-based authentication methods (e.g., passwords, certificates) are not well-suited for remote healthcare as patients could hand over credentials to someone else. Furthermore, one-time authentication using credentials or trait-based biometrics (e.g., face, fingerprints, iris) do not cover the entire monitoring period and may lead to unauthorized post-authentication use. Recent studies have shown that the human electrocardiogram (ECG) exhibits unique patterns that can be used to discriminate individuals. However, perturbation of the ECG signal due to physical activity is a major obstacle in applying the technology in real-world situations. In this paper, we present a novel ECG and accelerometer-based system that can authenticate individuals in an ongoing manner under various activity conditions. We describe the probabilistic authentication system we have developed and present experimental results from 17 individuals.

Top end mobile phones include a number of specialized (e.g., accelerometer, compass, GPS) and general purpose sensors (e.g., microphone, camera) that enable new people-centric sensing applications. Perhaps the most ubiquitous and unexploited sensor on mobile phones is the microphone - a powerful sensor that is capable of making sophisticated inferences about human activity, location, and social events from sound. In this paper, we exploit this untapped sensor not in the context of human communications but as an enabler of new sensing applications. We propose SoundSense, a scalable framework for modeling sound events on mobile phones. SoundSense is implemented on the Apple iPhone and represents the first general purpose sound sensing system specifically designed to work on resource limited phones. The architecture and algorithms are designed for scalability and Soundsense uses a combination of supervised and unsupervised learning techniques to classify both general sound types (e.g., music, voice) and discover novel sound events specific to individual users. The system runs solely on the mobile phone with no back-end interactions. Through implementation and evaluation of two proof of concept people-centric sensing applications, we demostrate that SoundSense is capable of recognizing meaningful sound events that occur in users' everyday lives.

We present an automated approach for studying fine-grained details of social interaction and relationships. Specifically, we analyze the conversational characteristics of a group of 24 individuals over a six-month period, explore the relationship between conversational dynamics and network position, and identify behavioral correlates of tie strengths within a network. The ability to study conversational dynamics and social networks over long time scales, and to investigate their interplay with rigor, objectivity, and transparency will complement the traditional methods for scientific inquiry into social dynamics. They may also enable socially aware ubiquitous computing systems that are cognizant of and responsive to the user's engagement with her social environment.

No abstract available.

The UW dynamic social network study is an effort to automatically observe and model the creation and evolution of a social network formed through spontaneous face-to-face conversations. We have collected more than 4,400 hours of data that capture the real world interactions between 24 subjects over a period of 9 months. The data was recorded in completely unconstrained and natural conditions, but was collected in a manner that protects the privacy of both study participants and non-participants. Despite the privacy constraints, the data allows for many different types of inference that are in turn useful for studying the prosodic and paralinguistic features of truly spontaneous speech across many subjects and over an extended period of time. This paper describes the new challenges and opportunities presented in such a study, our data collection effort, the problems we encountered, and the resulting corpus.

We propose an approach to activity recognition based on detecting and analyzing the sequence of objects that are being manipulated by the user. In domains such as cooking, where many activities involve similar actions, object-use information can be a valuable cue. In order for this approach to scale to many activities and objects, however, it is necessary to minimize the amount of human-labeled data that is required for modeling. We describe a method for automatically acquiring object models from video without any explicit human supervision. Our approach leverages sparse and noisy readings from RFID tagged objects, along with common-sense knowledge about which objects are likely to be used during a given activity, to bootstrap the learning process. We present a dynamic Bayesian network model which combines RFID and video data to jointly infer the most likely activity and object labels. We demonstrate that our approach can achieve activity recognition rates of more than 80% on a real-world dataset consisting of 16 household activities involving 33 objects with significant background clutter. We show that the combination of visual object recognition with RFID data is significantly more effective than the RFID sensor alone. Our work demonstrates that it is possible to automatically learn object models from video of household activities and employ these models for activity recognition, without requiring any explicit human labeling.

We present privacy-sensitive methods for (1) automatically finding multi-person conversations in spontaneous, situated speech data and (2) segmenting those conversations into speaker turns. The methods protect privacy through a feature set that is rich enough to capture conversational styles and dynamics, but not sufficient for reconstructing intelligible speech. Experimental results show that the conversation finding method outperforms earlier approaches and that the speaker segmentation method is a significant improvement to the only other known privacy-sensitive method for speaker segmentation. Index Terms: conversation modeling, speaker diarization, privacy, context-aware computing

We present a new and efficient semi-supervised training method for parameter estimation and feature selection in conditional random fields (CRFs). In real-world applications such as activity recognition, unlabeled sensor traces are relatively easy to obtain whereas labeled examples are expensive and tedious to collect. Furthermore, the ability to automatically select a small subset of discriminatory features from a large pool can be advantageous in terms of computational speed as well as accuracy. In this paper, we introduce the semi-supervised virtual evidence boosting (sVEB) algorithm for training CRFs -- a semi-supervised extension to the recently developed virtual evidence boosting (VEB) method for feature selection and parameter learning. Semi-supervised VEB takes advantage of the unlabeled data via minimum entropy regularization -- the objective function combines the unlabeled conditional entropy with labeled conditional pseudo-likelihood. The sVEB algorithm reduces the overall system cost as well as the human labeling cost required during training, which are both important considerations in building real world inference systems. In a set of experiments on synthetic data and real activity traces collected from wearable sensors, we illustrate that our algorithm benefits from both the use of unlabeled data and automatic feature selection, and outperforms other semi-supervised training approaches.

influences In this work, we quantitatively investigate the ways in which a given person the joint turn-taking behavior in a conversation. After collecting an auditory database of social interactions among a group of twenty-three people via wearable sensors (66 hours of data each over two weeks), we apply speech and conversation detection methods to the auditory streams. These methods automatically locate the conversations, determine their participants, and mark which participant was speaking when. We then model the joint turn-taking behavior as a Mixed-Memory Markov Model [1] that combines the statistics of the individual subjects' self-transitions and the partners ' cross-transitions. The mixture parameters in this model describe how much each person's individual behavior contributes to the joint turn-taking behavior of the pair. By estimating these parameters, we thus estimate how much influence each participant has in determining the joint turn(cid:173) this measure correlates taking behavior. We significantly with betweenness centrality [2], an independent measure of an individual's importance in a social network. This result suggests that our estimate of conversational influence is predictive of social influence.

Intelligent agents need to learn how the communication structure evolves within interacting groups and how to influence the groups overall behavior. We are developing methods to automatically and unobtrusively learn the social network structure that arises within a human group based on wearable sensors. Computational models of group interaction dynamics are derived from data gathered using wearable sensors. The questions we are exploring are: Can we tell who influences whom? Can we quantify this amount of influence? How can we modify group interactions to promote better information diffusion? The goal is real-time learning and modification of social network relationships by applying statistical machine learning techniques to data obtained from unobtrusive wearable sensors.

This paper presents motion field histograms as a new way of extracting facial features and modeling expressions. Features are based on local receptive field histograms, which are robust against errors in rotation, translation and scale changes during image alignment. Motion information is incorporated into the histograms by using difference images instead of raw images. We take the principal components of these histograms of selected facial regions and use the top 20 eigenvectors for compact representation. The eigen-coefficients are then used to model the temporal structure of different facial expressions from real-life data in the presence of translational and rotational errors that arise from head-tracking. The results demonstrate a 44% average performance increase over traditional optic flow methods for expressions extracted from unconstrained interactions.