The mindset people have about stress is important to be studied because this core belief, that stress is either enhancing or debilitating, fundamentally alters a person’s physiological and psychological responses to stressors. However, this crucial construct is rarely considered in prior research on momentary stress detection with wearables, leaving two fundamental questions unanswered: can wearable data identify an individual’s stress mindset, and can mindset be leveraged to build better performing stress detection models? To investigate that, we conducted an in-lab study with wearable devices by inducing mental stress in participants (N=23). First, we found that heart rate variability and electrodermal activity features carry signatures of stress mindset. Second, machine learning models can discriminate stress mindset with sensors, achieving AUCs upto 0.88. Finally, a random forest model trained for stress-is-enhancing participants outperformed a one-size-fits-all model (AUC=0.91 vs. 0.78, p < 0.05), for the task of stress detection. Our findings show that stress mindset leaves a measurable physiological footprint and that mindset-aware models open the potential for more personalized stress detection and interventions. To support future research, we publicly release the anonymized dataset at https://social-dynamics.net/stress/mindset

The interplay between mood and eating episodes has been extensively researched within the fields of nutrition, psychology, and behavioral science, revealing a connection between the two. Previous studies have relied on questionnaires and mobile phone self-reports to investigate the relationship between mood and eating. In more recent work, phone sensor data has been utilized to characterize both eating behavior and mood independently, particularly in the context of mobile food diaries and mobile health applications. However, current literature exhibits several limitations: a lack of investigation into the generalization of mood inference models trained with data from various everyday life situations to specific contexts like eating; an absence of studies using sensor data to explore the intersection of mood and eating; and inadequate examination of model personalization techniques within limited label settings, a common challenge in mood inference (i.e., far fewer negative mood reports compared to positive or neutral reports). In this study, we examined the everyday eating and mood using two separate datasets from two different studies: (i) Mexico (N \({}_{MEX}\) = 84, 1,843 mood-while-eating reports with a label distribution of positive: 51.7%, neutral: 38.6%, and negative: 9.8%) in 2019, and (ii) eight countries (N \({}_{MUL}\) = 678, 329K mood reports, including 24K mood-while-eating reports with a label distribution of positive: 83%, neutral: 14.9%, and negative: 2.2%) in 2020, which contain both passive smartphone sensing and self-report data. Our results indicate that generic mood inference models experience a decline in performance in specific contexts, such as during eating, highlighting the issue of sub-context shifts in mobile sensing. Moreover, we discovered that population-level (non-personalized) and hybrid (partially personalized) modeling techniques fall short in the commonly used three-class mood inference task (positive, neutral, negative). Additionally, we found that user-level modeling posed challenges for the majority of participants due to insufficient labels and data in the negative class. To overcome these limitations, we implemented a novel community-based personalization approach, building models with data from a set of users similar to the target user. Our findings demonstrate that mood-while-eating can be inferred with accuracies 63.8% (with F1 score of 62.5) for the MEX dataset and 88.3% (with F1 score of 85.7) with the MUL dataset using community-based models, surpassing those achieved with traditional methods.

Mental fatigue is a crucial aspect that has gained attention across various disciplines due to its impact on overall well-being. While previous research has explored the use of wearable devices for detecting mental fatigue, limited investigation has been conducted into the effectiveness of these devices in different body positions or in multi-device setups. To address this, our study utilizes a unique public dataset containing over 13 hours of sensor data collected across 36 sessions, with four wearable devices (Earable, Chestband, Wristband, and Headband). We propose several machine learning-based approaches to assess both psychological and physiological mental fatigue levels in a multimodal and multi-device environment. Specifically, we introduce device type-specific approaches (trained and tested on a single device) and multi-device approaches (trained and tested on multiple devices) for mental fatigue inference tasks. Our findings show that device type-specific models perform well, with AUC scores ranging from 0.63 to 0.69 for psychological and from 0.74 to 0.80 for physiological mental fatigue. The multi-device approach shows improved performance for psychological mental fatigue (AUC of 0.69 to 0.74) and physiological mental fatigue (AUC of 0.81 to 0.88). Hence, this study presents a unique and in-depth analysis of detecting mental fatigue with wearables, demonstrating the potential of machine learning-based approaches in multi-device and multimodal setups that are prevalent in today’s emerging lifestyles.

Understanding how social situations unfold in people’s daily lives is relevant to designing mobile systems that can support users in their personal goals, well-being, and activities. As an alternative to questionnaires, some studies have used passively collected smartphone sensor data to infer social context (i.e., being alone or not) with machine learning models. However, the few existing studies have focused on specific daily life occasions and limited geographic cohorts in one or two countries. This limits the understanding of how inference models work in terms of generalization to everyday life occasions and multiple countries. In this paper, we used a novel, large-scale, and multimodal smartphone sensing dataset with over 216K self-reports collected from 581 young adults in five countries (Mongolia, Italy, Denmark, UK, Paraguay), first to understand whether social context inference is feasible with sensor data, and then, to know how behavioral and country-level diversity affects inferences. We found that several sensors are informative of social context, that partially personalized multi-country models (trained and tested with data from all countries) and country-specific models (trained and tested within countries) can achieve similar performance above 90% AUC, and that models do not generalize well to unseen countries regardless of geographic proximity. These findings confirm the importance of the diversity of mobile data, to better understand social context inference models in different countries.

Machine learning models trained with passive sensor data from mobile devices can be used to perform various inferences pertaining to activity recognition, context awareness, and health and well-being. Prior work has improved inference performance through the use of multimodal sensors (inertial, GPS, proximity, app usage, etc.) or improved machine learning. In this context, a few studies shed light on critical issues relating to the poor cross-country generalization of models due to distributional shifts across countries. However, these studies have largely relied on inference performance as a means of studying generalization issues, failing to investigate whether the root cause of the problem is linked to specific sensor modalities (independent variables) or the target attribute (dependent variable). In this paper, we study this issue in complex activities of daily living (ADL) inference task, involving 12 classes, by using a multimodal, multi-country dataset collected from 689 participants across eight countries. We first show that the ‘country of origin’ of data is captured by sensors and can be inferred from each modality separately, with an average accuracy of 65%. We then propose two diversity scores (DS) that measure how a country differentiates from others w.r.t. sensor modalities or activities. Using these diversity scores, we observed that both individual sensor modalities and activities have the ability to differentiate countries. However, while many activities capture country differences, only the ‘App usage’ and ‘Location’ sensors can do so. By dissecting country-level diversity across dependent and independent variables, we provide a framework to better understand model generalization issues across countries and country-level diversity of sensing modalities.

Smartphones enable understanding human behavior with activity recognition to support people’s daily lives. Prior studies focused on using inertial sensors to detect simple activities (sitting, walking, running, etc.) and were mostly conducted in homogeneous populations within a country. However, people are more sedentary in the post-pandemic world with the prevalence of remote/hybrid work/study settings, making detecting simple activities less meaningful for context-aware applications. Hence, the understanding of (i) how multimodal smartphone sensors and machine learning models could be used to detect complex daily activities that can better inform about people’s daily lives, and (ii) how models generalize to unseen countries, is limited. We analyzed in-the-wild smartphone data and ∼ 216K self-reports from 637 college students in five countries (Italy, Mongolia, UK, Denmark, Paraguay). Then, we defined a 12-class complex daily activity recognition task and evaluated the performance with different approaches. We found that even though the generic multi-country approach provided an AUROC of 0.70, the country-specific approach performed better with AUROC scores in [0.79-0.89]. We believe that research along the lines of diversity awareness is fundamental for advancing human behavior understanding through smartphones and machine learning, for more real-world utility across countries.

Being able to assess dog personality can be used to, for example, match shelter dogs with future owners, and personalize dog activities. Such an assessment typically relies on experts or psychological scales administered to dog owners, both of which are costly. To tackle that challenge, we built a device called “Patchkeeper” that can be strapped on the pet’s chest and measures activity through an accelerometer and a gyroscope. In an in-the-wild deployment involving 12 healthy dogs, we collected 1300 hours of sensor activity data and dog personality test results from two validated questionnaires. By matching these two datasets, we trained ten machine learning classifiers that predicted dog personality from activity data, achieving AUCs in [0.63-0.90], suggesting the value of tracking psychological signals of pets using wearable technologies.

Diversity-aware platform design is a paradigm that responds to the ethical challenges of existing social media platforms. Available platforms have been criticized for minimizing users' autonomy, marginalizing minorities, and exploiting users' data for profit maximization. This paper presents a design solution that centers the well-being of users. It presents the theory and practice of designing a diversity-aware platform for social relations. In this approach, the diversity of users is leveraged in a way that allows like-minded individuals to pursue similar interests or diverse individuals to complement each other in a complex activity. The end users of the envisioned platform are students, who participate in the design process. Diversity-aware platform design involves numerous steps, of which two are highlighted in this paper: 1) defining a framework and operationalizing the "diversity" of students, 2) collecting "diversity" data to build diversity-aware algorithms. The paper further reflects on the ethical challenges encountered during the design of a diversity-aware platform.

In mobile crowd-sourcing systems, simply relying on people to opportunistically select and perform tasks typically leads to drawbacks such as low task acceptance/completion rates and undesirable spatial skews. In this paper, we utilize data from TASKer, a campus-based mobile crowd-sourcing platform, to empirically study and discover whether and how various context-aware notification strategies can help overcome such drawbacks. We first study worker interactions, in the absence of any notifications, to discover some spatio-temporal properties of task acceptance and completion. Based on these insights, we then experimentally demonstrate the effectiveness of two novel, non-personal, context-driven notification strategies, comparing the outcomes to two different baselines (no-notification and random-notification). Finally, using the data from the random-notification mechanism, we derive a classification model, incorporating several novel contextual features, that can predict a worker's responsiveness to notifications with high accuracy. Our work extends the crowd-sourcing literature by emphasizing the power of smart notifications for greater worker engagement.

Natural human interactions for Mixed Reality Applications are overwhelmingly multimodal: humans communicate intent and instructions via a combination of visual, aural and gestural cues. However, supporting low-latency and accurate comprehension of such multimodal instructions (MMI), on resource-constrained wearable devices, remains an open challenge, especially as the state-of-the-art comprehension techniques for each individual modality increasingly utilize complex Deep Neural Network models. We demonstrate the possibility of overcoming the core limitation of latency-vs.-accuracy tradeoff by exploiting cross-modal dependencies-i.e., by compensating for the inferior performance of one model with an increased accuracy of more complex model of a different modality. We present a sensor fusion architecture that performs MMI comprehension in a quasi-synchronous fashion, by fusing visual, speech and gestural input. The architecture is reconfigurable and supports dynamic modification of the complexity of the data processing pipeline for each individual modality in response to contextual changes. Using a representative “classroom” context and a set of four common interaction primitives, we then demonstrate how the choices between low and high complexity models for each individual modality are coupled. In particular, we show that (a) a judicious combination of low and high complexity models across modalities can offer a dramatic 3-fold decrease in comprehension latency together with an increase ~10-15% in accuracy, and (b) the right collective choice of models is context dependent, with the performance of some model combinations being significantly more sensitive to changes in scene context or choice of interaction.

Understanding food consumption patterns and contexts using mobile sensing is fundamental to build mobile health applications that require minimal user interaction to generate mobile food diaries. Many available mobile food diaries, both commercial and in research, heavily rely on self-reports, and this dependency limits the long term adoption of these apps by people. The social context of eating (alone, with friends, with family, with a partner, etc.) is an important self-reported feature that influences aspects such as food type, psychological state while eating, and the amount of food, according to prior research in nutrition and behavioral sciences. In this work, we use two datasets regarding the everyday eating behavior of college students in two countries, namely Switzerland (Nch=122) and Mexico (Nmx=84), to examine the relation between the social context of eating and passive sensing data from wearables and smartphones. Moreover, we design a classification task, namely inferring eating-alone vs. eating-with-others episodes using passive sensing data and time of eating, obtaining accuracies between 77% and 81%. We believe that this is a first step towards understanding more complex social contexts related to food consumption using mobile sensing.

Smartphone applications that use passive sensing to support human health and well-being primarily rely on: (a) generating low-dimensional representations from high-dimensional data streams; (b) making inferences regarding user behavior; and (c) using those inferences to benefit application users. Meanwhile, sometimes these datasets are shared with third parties as well. Human-centered ubiquitous systems need to ensure that sensitive attributes of users are protected when applications provide utility to people based on such behavioral inferences. In this paper, we demonstrate that inferences of sensitive attributes of users (gender, body mass index category) are possible using low-dimensional and sparse data coming from mobile food diaries (a combination of sensor data and self-reports). After exposing this potential risk, we demonstrate how deep learning techniques can be used for feature transformation to preserve sensitive user information while achieving high accuracies for application-related inferences (e.g. inferring the type of consumed food). Our work is based on two datasets of daily eating behavior of 160 young adults from Switzerland (NCH=122) and Mexico (NMX=38). Results show that using the proposed approach, accuracies in the order of 75%-90% can be achieved for application related inferences, while reducing the sensitive inference to almost random performance.

While analysis of urban commuting data has a long and demonstrated history of providing useful insights into human mobility behavior, such analysis has been performed largely in offline fashion and to aid medium-to-long term urban planning. In this work, we demonstrate the power of applying predictive analytics on real-time mobility data, specifically the smart-card generated trip data of millions of public bus commuters in Singapore, to create two novel and "live" smart city services. The key analytical novelty in our work lies in combining two aspects of urban mobility: (a) conformity: which reflects the predictability in the aggregated flow of commuters along bus routes, and (b) regularity: which captures the repeated trip patterns of each individual commuter. We demonstrate that the fusion of these two measures of behavior can be performed at city-scale using our BuScope platform, and can be used to create two innovative smart city applications. The Last-Mile Demand Generator provides O(mins) lookahead into the number of disembarking passengers at neighborhood bus stops; it achieves over 85% accuracy in predicting such disembarkations by an ingenious combination of individual-level regularity with aggregate-level conformity. By moving driverless vehicles proactively to match this predicted demand, we can reduce wait times for disembarking passengers by over 75%. Independently, the Neighborhood Event Detector uses outlier measures of currently operating buses to detect and spatiotemporally localize dynamic urban events, as much as 1.5 hours in advance, with a localization error of ~450 meters.

Digital documents are likely to have problems associated with the persistence of links, especially when dealing with references to external resources. People keep track of various webpages of their interest using distributed digital collections and without possession of these documents; the curator cannot control how they change. In the current context, managing these distributed digital collections and getting notifications about various changes have become a significant challenge. In this paper, we address the architectural aspects of change detection systems and present optimized change detection architecture, including a web service and a browser plugin, along with an email notification service. We have performed an experimental study on our hybrid architecture for change detection in a distributed digital collection. The proposed method introduces a preliminary framework that can serve as a useful tool to mitigate the impact of unexpected change in documents stored in decentralized collections in the future.