College students every day decide and plan how to best spend their time to balance academic, physical, and social goals under uncertainty. This process is likely suboptimal where long-term life satisfaction and success is not guaranteed, and poor decision-making may lead to longer-term problems like depression. To support everyday planning, we introduce activity recommendation, a novel method that combines artificial intelligence, machine learning, and a psychology-informed approach to automatically generate activity-recommendations that optimize long-term life satisfaction. We tested our method with an existing dataset and derived activity recommendations for depressed and non-depressed students. We evaluated the recommendations through interviews with college students who rated the suggestions positively. Our model can be optimized for different goals and domains and is easy to interpret. Our results demonstrate the feasibility of our approach and lay the groundwork towards implementing a live system.

An ability to detect behaviors that negatively impact people's wellbeing and show people how they can correct those behaviors could enable technology that improves people's lives. Existing supervised machine learning approaches to detect and generate such behaviors require lengthy and expensive data labeling by domain experts. In this work, we focus on the domain of routine behaviors, where we model routines as a series of frequent actions that people perform in specific situations. We present an approach that bypasses labeling each behavior instance that a person exhibits. Instead, we weakly label instances using people's demonstrated routine. We classify and generate new instances based on the probability that they belong to the routine model. We illustrate our approach on an example system that helps drivers become aware of and understand their aggressive driving behaviors. Our work enables technology that can trigger interventions and help people reflect on their behaviors when those behaviors are likely to negatively impact them.

Previous work on muscle activity sensing has leveraged specialized sensors such as electromyography and force sensitive resistors. While these sensors show great potential for detecting finger/hand gestures, they require additional hardware that adds to the cost and user discomfort. Past research has utilized sensors on commercial devices, focusing on recognizing gross hand gestures. In this work we present Serendipity, a new technique for recognizing unremarkable and fine-motor finger gestures using integrated motion sensors (accelerometer and gyroscope) in off-the-shelf smartwatches. Our system demonstrates the potential to distinguish 5 fine-motor gestures like pinching, tapping and rubbing fingers with an average f1-score of 87%. Our work is the first to explore the feasibility of using solely motion sensors on everyday wearable devices to detect fine-grained gestures. This promising technology can be deployed today on current smartwatches and has the potential to be applied to cross-device interactions, or as a tool for research in fields involving finger and hand motion.

Understanding smartphone users is fundamental for creating better smartphones, and improving the smartphone usage experience and generating generalizable and reproducible research. However, smartphone manufacturers and most of the mobile computing research community make a simplifying assumption that all smartphone users are similar or, at best, constitute a small number of user types, based on their behaviors. Manufacturers design phones for the broadest audience and hope they work for all users. Researchers mostly analyze data from smartphone-based user studies and report results without accounting for the many different groups of people that make up the user base of smartphones. In this work, we challenge these elementary characterizations of smartphone users and show evidence of the existence of a much more diverse set of users. We analyzed one month of application usage from 106,762 Android users and discovered 382 distinct types of users based on their application usage behaviors, using our own two-step clustering and feature ranking selection approach. Our results have profound implications on the reproducibility and reliability of mobile computing studies, design and development of applications, determination of which apps should be pre-installed on a smartphone and, in general, on the smartphone usage experience for different types of users.

Activity recognition is a key component of context-aware computing to support people's physical activity, but conventional approaches often lack in their generalizability and scalability due to problems of diversity in how individuals perform activities, overfitting when building activity models, and collection of a large amount of labeled data from end users. To address these limitations, we propose a semipopulation-based approach that exploits activity models trained from other users; therefore, a new user does not need to provide a large volume of labeled activity data. Instead of relying on any additional information from users like their weight or height, our approach directly measures the fitness of others' models on a small amount of labeled data collected from the new user. With these shared activity models among users, we compose a hybrid model of Bayesian networks and support vector machines to accurately recognize the activity of the new user. On activity data collected from 28 people with a diversity in gender, age, weight, and height, our approach produced an average accuracy of 83.4% (kappa: 0.852), compared with individual and (standard) population models that had accuracies of 77.3% (kappa: 0.79) and 77.7% (kappa: 0.743), respectively. Through an analysis on the performance of our approach and users' demographic information, our approach outperforms others that rely on users' demographic information for recognizing their activities, which may contradict the commonly held belief that physically similar people would have similar activity patterns.

In today's ubiquitous computing environment where users carry, manipulate, and interact with an increasing number of networked devices, applications and web services, human attention is the new bottleneck in computing. It is therefore important to minimize a user's mental effort due to notifications, especially in situations where users are mobile and using multiple wearable and mobile devices. To this end, we propose Attelia II, a novel middleware that identifies breakpoints in users' lives while using those devices, and delivers notifications at these moments. Attelia II works in real-time and uses only the mobile and wearable devices that users naturally use and wear, without any modifications to applications, and without any dedicated psycho-physiological sensors. Our in-the-wild evaluation in users' multi-device environment (smart phones and smart watches) with 41 participants for 1 month validated the effectiveness of Attelia. Our new physical activity-based breakpoint detection, in addition to the UI Event-based breakpoint detection, resulted in a 71.8% greater reduction of users' perception of workload, compared with our previous system that used UI events only. Adding this functionality to a smart watch reduced workload perception by 19.4% compared to random timing of notification deliveries. Our multi-device breakpoint detection across smart phones and watches resulted in about 3 times greater reduction in workload perception than our previous system.

In today's ubiquitous computing environment where the number of devices, applications and web services are ever increasing, human attention is the new bottleneck in computing. To minimize user cognitive load, we propose Attelia, a novel middleware that identifies breakpoints in user interaction and delivers notifications at these moments. Attelia works in realtime and uses only the mobile devices that users naturally use and wear, without any modifications to applications, and without any dedicated psycho-physiological sensors. Our evaluation proved the effectiveness of Attelia. A controlled user study showed that notifications at detected breakpoint timing resulted in 46% lower cognitive load compared to randomly-timed notifications. Furthermore, our “in-the-wild” user study with 30 participants for 16 days further validated Attelia's value, with a 33% decrease in cognitive load compared to randomly-timed notifications.

With advances in physiological sensors, we are able to understand people's physiological status and recognize stress to provide beneficial services. Despite the great potential in physiological stress recognition, there are some critical issues that need to be addressed such as the sensitivity and variability of physiology to many factors other than stress (e.g., physical activity). To resolve these issues, in this paper, we focus on the understanding of physiological responses to both stressor and physical activity and perform stress recognition, particularly in situations having multiple stimuli: physical activity and stressors. We construct stress models that correspond to individual situations, and we validate our stress modeling in the presence of physical activity. Analysis of our experiments provides an understanding on how physiological responses change with different stressors and how physical activity confounds stress recognition with physiological responses. In both objective and subjective settings, the accuracy of stress recognition drops by more than 14% when physical activity is performed. However, by modularizing stress models with respect to physical activity, we can recognize stress with accuracies of 82% (objective stress) and 87% (subjective stress), achieving more than a 5-10% improvement from approaches that do not take physical activity into account.

Much research in ubiquitous computing assumes that a user's phone will be always on and at-hand, for collecting user context and for communicating with a user. Previous work with the previous generation of mobile phones has shown that such an assumption is false. Here, we investigate whether this assumption about users' proximity to their mobile phones holds for a new generation of mobile phones, smart phones. We conduct a data collection field study of 28 smart phone owners over a period of 4 weeks. We show that in fact this assumption is still false, with the within arm's reach proximity being true close to 50% of the time, similar to the earlier work. However, we also show that smart phone proximity within the same room (arm+room) as the user is true almost 90% of the time. We discuss the reasons for these phone proximities and the implications of this on the development of mobile phone applications, particularly those that collect user and environmental context, and delivering notification to users. We also show that we can accurately predict the proximity at the arm level and arm+room level with 75 and 83% accuracy, respectively, with features simple to collect and model on a mobile phone. Further we show that for several individuals who are almost always within the arm+room level, we can predict this level with over 90% accuracy.

In this paper we present a novel scheme for unstructured audio scene classification that possesses three highly desirable and powerful features: autonomy, scalability, and robustness. Our scheme is based on our recently introduced machine learning algorithm called Simultaneous Temporal And Contextual Splitting (STACS) that discovers the appropriate number of states and efficiently learns accurate Hidden Markov Model (HMM) parameters for the given data. STACS-based algorithms train HMMs up to five times faster than Baum-Welch, avoid the overfitting problem commonly encountered in learning large state-space HMMs using Expectation Maximization (EM) methods such as Baum-Welch, and achieve superior classification results on a very diverse dataset with minimal pre-processing. Furthermore, our scheme has proven to be highly effective for building real-world applications and has been integrated into a commercial surveillance system as an event detection component.