Abstract Powered by smartphones and wearable devices, the Experience Sampling Method (ESM) has increased in popularity for studying behaviors, thoughts, and experiences over time and in situ. Participants in ESM studies receive several notifications a day to self-report but often disengage due to intrusive and poorly timed notifications. Consequently, the response rate drops over time, hampering data collection and degrading ecological validity. Researchers have experimented with various strategies to optimize notification scheduling, including personalization, context sensing, and machine learning (ML). Edge computing can facilitate the training of ML models without the need for server communications, which is especially convenient for in-the-wild studies with unreliable network connectivity. Complementary logical evaluations on edge devices can minimize participant burden by accounting for sampling density, i.e., ensuring a minimum number of well-distributed daily notifications. However, these efforts raise engineering and scientific challenges related to avoiding cold start and training models on smartwatches. To overcome these challenges, we propose an open-source architecture and software that facilitates online learning to optimize notification delivery. Our feasibility study with $$N=37$$ N = 37 participants resulted in a response rate of 10.2% higher and a reaction time of 9.6% lower on average compared to the classical interval-based sampling.

The Experience Sampling Method (ESM) is commonly used to understand behaviors, thoughts, and feelings in the wild by collecting self-reports. Sustaining sufficient response rates, especially in long-running studies remains challenging. To avoid low response rates and dropouts, experimenters rely on their experience, proposed methodologies from earlier studies, trial and error, or the scarcely available participant behavior data from previous ESM protocols. This approach often fails in finding the acceptable study parameters, resulting in redesigning the protocol and repeating the experiment. Research has shown the potential of machine learning to personalize ESM protocols such that ESM prompts are delivered at opportune moments, leading to higher response rates. The corresponding training process is hindered due to the scarcity of open data in the ESM domain, causing a cold start, which could be mitigated by simulating participant behavior. Such simulations provide training data and insights for the experimenters to update their study design choices. Creating this simulation requires behavioral science, psychology, and programming expertise. Large language models (LLMs) have emerged as facilitators for information inquiry and programming, albeit random and occasionally unreliable. We aspire to assess the readiness of LLMs in an ESM use case. We conducted research using GPT-3.5 turbo-16k to tackle an ESM simulation problem. We explored several prompt design alternatives to generate ESM simulation programs, evaluated the output code in terms of semantics and syntax, and interviewed ESM practitioners. We found that engineering LLM-enabled ESM simulations have the potential to facilitate data generation, but they perpetuate trust and reliability challenges.

Smartwatches are emerging as an increasingly popular platform for longitudinal in situ data collection with methods often referred to as experience sampling and ecological momentary assessment. Their small size challenges designers of relevant applications to ensure usability and a positive user experience. This paper investigates the usability of different input techniques for responding to in situ surveys administered on smartwatches. In this paper, we classify different input techniques that can support this task. Then, we report on two user studies that compared different input techniques and their suitability at two levels of user activity: while sitting and while walking. A pilot study (N = 18) examined numeric input with three input techniques that utilize common features of smartwatches with a touchscreen: Multi-Step Tapping, Bezel Rotation, and Swiping. The main study (N = 80) examined numeric input and list selection including in the comparison two more techniques: Long-List Tapping and Virtual Buttons to scroll through options. Overall, we found that whether users are seated or walking did not affect the speed or accuracy of input. Bezel rotation was the slowest input technique but also the most accurate. Swiping resulted in most errors. Long-List Tapping yielded the shortest reaction times. Future research should examine different form factors for the smartwatch and diverse usage contexts.