Clinical pre-consultation, where patients share health information prior to an appointment, can offer better patient-centered care by freeing time for meaningful patient–physician conversations. Conversational agents powered by large language models (LLMs) can automate this process to make it more scalable and consistent, but doing so can cause information overload that exacerbates physicians’ workload as they spend time parsing through data. This paper examines the opportunities and challenges of using pre-consultation agents to mediate information transfer between patients and physicians, with the aim of producing clinically useful yet patient-driven summaries that capture their histories and concerns. Through sessions with both patients and physicians, we show that automatically generated pre-consultation summaries can increase patients’ confidence and sense of control over their health information while fostering a more collaborative dynamic. We conclude with design recommendations for integrating pre-consultation agents into clinical workflows.

Continuous speech monitoring using smartphones and smartwatches offers numerous opportunities to detect and predict health deterioration. However, the algorithms used to deliver these benefits must account for the realities of periodic audio recording in the wild, including other voices and overlapping noise. To address the former without clean sample voice recordings, we use speaker diarization to identify the person who speaks the most throughout a given day. To address the latter without compromising the validity of extracted speech features, we leverage the intuition that a reliable feature should remain unaltered by a noise-reduction technique when background noise is minimal. We examine both techniques in the context of a longitudinal dataset collected from 16 patients with chronic obstructive pulmonary disease (COPD) over roughly 3 months. Our best model achieved an AUROC of 0.86 when predicting the presence of respiratory symptoms, outperforming baseline models that rely solely on voice activity detection and noise reduction.

Stories about overcoming personal struggles can effectively illustrate the application of psychological theories in real life, yet they may fail to resonate with individuals' experiences. In this work, we employ large language models (LLMs) to create tailored narratives that acknowledge and address unique challenging thoughts and situations faced by individuals. Our study, involving 346 young adults across two settings, demonstrates that personalized LLM-enhanced stories were perceived to be better than human-written ones in conveying key takeaways, promoting reflection, and reducing belief in negative thoughts. These stories were not only seen as more relatable but also similarly authentic to human-written ones, highlighting the potential of LLMs in helping young adults manage their struggles. The findings of this work provide crucial design considerations for future narrative-based digital mental health interventions, such as the need to maintain relatability without veering into implausibility and refining the wording and tone of AI-enhanced content.

Techniques that use speech analysis for tasks like health monitoring and emotion recognition usually operate on moderately sized windows with little regard for what the individual is saying. In this work, we argue that isolating specific phonemes within speech offers greater nuance that leads to more consistent yet natural sounds for analysis. We examine this hypothesis in the context of lung function estimation. We recruited 11 patients with chronic obstructive pulmonary disease (COPD) to read from a script and perform spirometry to quantify their lung function. After segmenting their audio recordings into discrete phonemes, we extracted various phonation, prosodic, and spectral features to summarize their acoustic qualities. We then examined the correlation between those audio features and measurements from spirometry, observing that certain combinations of features and phonemes led to higher correlations than the best-performing phoneme-agnostic baseline for our dataset.

Challenges in engagement with digital mental health (DMH) tools are commonly addressed through technical enhancements and algorithmic interventions. This paper shifts the focus towards the role of users' broader social context as a significant factor in engagement. Through an eight-week text messaging program aimed at enhancing psychological wellbeing, we recruited 20 participants to help us identify situational engagement disruptors (SEDs), including personal responsibilities, professional obligations, and unexpected health issues. In follow-up design workshops with 25 participants, we explored potential solutions that address such SEDs: prioritizing self-care through structured goal-setting, alternative framings for disengagement, and utilization of external resources. Our findings challenge conventional perspectives on engagement and offer actionable design implications for future DMH tools.

Traditional interventions for academic procrastination often fail to capture the nuanced, individual-specific factors that underlie them. Large language models (LLMs) hold immense potential for addressing this gap by permitting open-ended inputs, including the ability to customize interventions to individuals' unique needs. However, user expectations and potential limitations of LLMs in this context remain underexplored. To address this, we conducted interviews and focus group discussions with 15 university students and 6 experts, during which a technology probe for generating personalized advice for managing procrastination was presented. Our results highlight the necessity for LLMs to provide structured, deadline-oriented steps and enhanced user support mechanisms. Additionally, our results surface the need for an adaptive approach to questioning based on factors like busyness. These findings offer crucial design implications for the development of LLM-based tools for managing procrastination while cautioning the use of LLMs for therapeutic guidance.

Silent speech interaction (SSI) allows users to discreetly input text without using their hands. Existing wearable SSI systems typically require custom devices and are limited to a small lexicon, limiting their utility to a small set of command words. This work proposes ReHEarSSE, an earbud-based ultrasonic SSI system capable of generalizing to words that do not appear in its training dataset, providing support for nearly an entire dictionary’s worth of words. As a user silently spells words, ReHEarSSE uses autoregressive features to identify subtle changes in ear canal shape. ReHEarSSE infers words using a deep learning model trained to optimize connectionist temporal classification (CTC) loss with an intermediate embedding that accounts for different letters and transitions between them. We find that ReHEarSSE recognizes 100 unseen words with an accuracy of 89.3%.

Pre-consultation serves as a critical information exchange between healthcare providers and patients, streamlining visits and supporting patient-centered care. Human-led pre-consultations offer many benefits, yet they require significant time and energy from clinical staff. In this work, we identify design goals for pre-consultation chatbots given their potential to carry out human-like conversations and autonomously adapt their line of questioning. We conducted a study with 33 walk-in clinic patients to elicit design considerations for pre-consultation chatbots. Participants were exposed to one of two study conditions: an LLM-powered AI agent and a Wizard-of-Oz agent simulated by medical professionals. Our study found that both conditions were equally well-received and demonstrated comparable conversational capabilities. However, the extent of the follow-up questions and the amount of empathy impacted the chatbot’s perceived thoroughness and sincerity. Patients also highlighted the importance of setting expectations for the chatbot before and after the pre-consultation experience.

Remote patient monitoring is becoming increasingly instrumental to healthcare delivery but can substantially hamper the interpersonal communication that underlies standard clinical practice. In this work, we explore the benefits imparted to patients, clinicians, and researchers by an asynchronous messaging feature within a platform called COVIDFree@Home. We created COVIDFree@Home to assist the healthcare system in a large metropolitan city in North America during the COVID-19 pandemic. Clinicians used COVIDFree@Home to monitor the self-reported symptoms and vital signs of over 350 COVID-19 patients post-infection. Using thematic analysis of user-initiated messages, we found the messaging feature helped maintain protocol adherence while allowing patients to ask questions about their health and clinicians to convey empathetic care. This feedback cycle also led to higher quality data for hospitalization prediction, as the revisions significantly improved the AUROC of a machine learning model trained on demographic variables, vital signs data, and self-reported symptoms from 0.53 to 0.59.

Menstrual trackers currently lack the affordances required to help individuals achieve their goals beyond menstrual event predictions and symptom logging. Taking an initial step towards this aspiration, we propose, validate, and refine five functional design requirements for future interface designs that facilitate menstrual data exploration. We interviewed 30 individuals who menstruate and collected their feedback on the practical application of these requirements. To elicit ideas and impressions, we designed two proof-of-concept interfaces to use as design probes with similar core functionalities but different presentations of phase timing predictions and signal arrangement. Our analysis revealed participants’ feedback regarding the presentation of predictions for menstrual-related events, the visualization of future signal patterns, personalization abilities for viewing signals relevant to their menstrual experience, the availability of resources to understand the underlying biological connections between signals, and the ability to compare multiple cycles side-by-side with context.

The need for frequent and brief practice in deep breathing presents challenges in maintaining motivation and consistency. While persuasive technologies have been shown to improve engagement in therapeutic exercises, there is a lack of insight into specific motivational strategies for such intermittent activities. We investigate how idle games can incentivize behaviors like deep breathing and identify specific mechanics for fostering an optimal practice cycle. We illustrate this approach in a game called BreathPurr-suade. After validating the physiological efficacy of the embedded breathing guide, our four-week study revealed idle games are more effective in maintaining deep breathing adherence than a standard breathing guide. Our work highlights the capacity of idle games to foster deep breathing, revealing their efficacy in subtle persuasive game designs that encourage intermittent therapeutic practices.

Poor quality sleep can be characterized by the occurrence of events ranging from body movement to breathing impairment. Widely available earbuds equipped with sensors (also known as earables) can be combined with a sleep event detection algorithm to offer a convenient alternative to laborious clinical tests for individuals suffering from sleep disorders. Although various solutions utilizing such devices have been proposed to detect sleep events, they ignore the fact that individuals often share sleeping spaces with roommates or couples. To address this issue, we introduce DreamCatcher, the first publicly available dataset for wearer-aware sleep event algorithm development on earables. DreamCatcher encompasses eight distinct sleep events, including synchronous dual-channel audio and motion data collected from 12 pairs (24 participants) totaling 210 hours (420 hour.person) with fine-grained label. We tested multiple benchmark models on three tasks related to sleep event detection, demonstrating the usability and unique challenge of DreamCatcher. We hope that the proposed DreamCatcher can inspire other researchers to further explore efficient wearer-aware human vocal activity sensing on earables. DreamCatcher is publicly available at https://github.com/thuhci/DreamCatcher.

Without a nuanced understanding of users' perspectives and contexts, text messaging tools for supporting psychological wellbeing risk delivering interventions that are mismatched to users' dynamic needs. We investigated the contextual factors that influence young adults' day-to-day experiences when interacting with such tools. Through interviews and focus group discussions with 36 participants, we identified that people's daily schedules and affective states were dominant factors that shape their messaging preferences. We developed two messaging dialogues centered around these factors, which we deployed to 42 participants to test and extend our initial understanding of users' needs. Across both studies, participants provided diverse opinions of how they could be best supported by messages, particularly around when to engage users in more passive versus active ways. They also proposed ways of adjusting message length and content during periods of low mood. Our findings provide design implications and opportunities for context-aware mental health management systems.

Medical consultations over synchronous text-based platforms are becoming increasingly popular for virtual care, yet little is known about how physicians translate their training to this healthcare medium. We report the constraints, workarounds, and opportunities highlighted by eight primary care physicians who used such a platform in simulated medical scenarios with standardized patients. We found that due to the perceived inefficiency of communicating over text, the physicians made subconscious use of double-barreled questions and action multiplexing to streamline the conversation. In addition, the physicians overcame the lack of missing verbal and visual cues by adding explicit messages to convey empathy and active listening. We also identify several affordances of text-based platforms, such as the ability for users to reference the conversation history and for patients to feel a sense of privacy during sensitive disclosure. From these findings, we propose design opportunities for how future synchronous text-based platforms can better support medical consultations.

One-way text messaging services have the potential to support psychological wellbeing at scale without conversational partners. However, there is limited understanding of what challenges are faced in mapping interactions typically done face-to-face or via online interactive resources into a text messaging medium. To explore this design space, we developed seven text messages inspired by cognitive behavioral therapy. We then conducted an open-ended survey with 788 undergraduate students and follow-up interviews with students and clinical psychologists to understand how people perceived these messages and the factors they anticipated would drive their engagement. We leveraged those insights to revise our messages, after which we deployed our messages via a technology probe to 11 students for two weeks. Through our mixed-methods approach, we highlight challenges and opportunities for future text messaging services, such as the importance of concrete suggestions and flexible pre-scheduled message timing.

Adopting new psychological strategies to improve mental wellness can be challenging since people are often unable to anticipate how new habits are applicable to their circumstances. Narrative-based interventions have the potential to alleviate this burden by illustrating psychological principles in an applied context. In this work, we explore how stories can be delivered via the ubiquitous and scalable medium of text messaging. Through formative work consisting of interviews and focus group discussions with 15 participants, we identified desirable elements of stories about mental health, including authenticity and relatability. We then deployed story-based text messages to 42 participants to explore challenges regarding both the stories' content (e.g., specific versus generalized) and format (e.g., story length). We observed that our stories helped participants reflect on and identify flaws in their thinking patterns. Our findings highlight design implications and opportunities for mental wellness interventions that utilize stories in text messaging services.

Poultry farming is a significant income-generating activity in sub-Saharan African (SSA) households. Poultry farmers frequently have to overcome extreme environmental conditions to maintain their chickens’ wellbeing. Prior research has proposed automating poultry farming activities to control environmental conditions (e.g., temperature and humidity). However, these interventions have never been implemented, in this context, to understand how they would work and participants’ perceptions. Further, chicken coops in SSA have different configurations that would make technology automation difficult. To explore how technology can be used to address this problem, we worked with local collaborators to design and deploy “NkhukuProbe”—a low-cost sensor-based technology that poultry farmers can interact with via USSD (Unstructured Supplementary Service Data) to monitor and adjust chicken coop conditions. First, we conducted a review of related work on poultry farming in SSA and a pilot study with poultry farming experts. Findings from this work guided the design of NkhukuProbe. Then, we deployed NkhukuProbe in 15 Malawian households for one month. The goals of our deployment were to understand participants’ experiences using NkhukuProbe and to learn about other ways of using sensors in this context. To achieve these goals, we used interview, diary, observation and data logging to collect data throughout the deployment. Our findings suggest that a technology probe's approach unveiled different opportunities for using sensors to support poultry farming in SSA. Further, NkhukuProbe motivated participants to think of other ways of using sensors. We present design implications based on these findings and offer new perspectives on the role of technology in supporting poultry farming activities.

Sleep is critical to human function, mediating factors like memory, mood, energy, and alertness; therefore, it is commonly conjectured that a good night’s sleep is important for job performance. However, both real-world sleep behavior and job performance are difficult to measure at scale. In this work, we demonstrate that people’s everyday interactions with online mobile apps can reveal insights into their job performance in real-world contexts. We present an observational study in which we objectively tracked the sleep behavior and job performance of salespeople (N = 15) and athletes (N = 19) for 18 months, leveraging a mattress sensor and online mobile app to conduct the largest study of this kind to date. We first demonstrate that cumulative sleep measures are significantly correlated with job performance metrics, showing that an hour of daily sleep loss for a week was associated with a 9.0% average reduction in contracts established for salespeople and a 9.5% average reduction in game grade for the athletes. We then investigate the utility of online app interaction time as a passively collectible and scalable performance indicator. We show that app interaction time is correlated with the job performance of the athletes, but not the salespeople. To support that our app-based performance indicator truly captures meaningful variation in psychomotor function as it relates to sleep and is robust against potential confounds, we conducted a second study to evaluate the relationship between sleep behavior and app interaction time in a cohort of 274 participants. Using a generalized additive model to control for per-participant random effects, we demonstrate that participants who lost one hour of daily sleep for a week exhibited average app interaction times that were 5.0% slower. We also find that app interaction time exhibits meaningful chronobiologically consistent correlations with sleep history, time awake, and circadian rhythms. The findings from this work reveal an opportunity for online app developers to generate new insights regarding cognition and productivity.

Year-round ultraviolet exposure silently causes skin damage that goes unnoticed until sunburn. Current personal wearables for monitoring UV exposure have not seen significant uptake, which may be attributed to their one-size-fits-all aesthetic or inapplicability to people with different skin tones. We present EcoPatches, inkjet-printable chemical patches that mediate a person's relationship with their environment by allowing them to create designs and formulations that resonate with them. Supporting human- and machine-interpretability for EcoPatches' visual changes means that users can glance at their EcoPatch during the day to see large exposure changes or take a picture of their EcoPatch with a smartphone app for more accurate and precise readings. We conducted an online survey to elicit visual design recommendations that support these features. We also evaluated both interpretation methods, finding that they achieved strong Pearson correlation coefficients with the \projectnames' known exposure levels (human: 0.79, app: 0.90).

Past research regarding on-body interaction typically requires custom sensors, limiting their scalability and generalizability. We propose EarBuddy, a real-time system that leverages the microphone in commercial wireless earbuds to detect tapping and sliding gestures near the face and ears. We develop a design space to generate 27 valid gestures and conducted a user study (N=16) to select the eight gestures that were optimal for both human preference and microphone detectability. We collected a dataset on those eight gestures (N=20) and trained deep learning models for gesture detection and classification. Our optimized classifier achieved an accuracy of 95.3%. Finally, we conducted a user study (N=12) to evaluate EarBuddy's usability. Our results show that EarBuddy can facilitate novel interaction and that users feel very positively about the system. EarBuddy provides a new eyes-free, socially acceptable input method that is compatible with commercial wireless earbuds and has the potential for scalability and generalizability

Rapid diagnostic tests (RDTs) provide point-of-care medical diagnosis without sophisticated laboratory equipment, making them especially useful for community health workers (CHWs). Because the procedure for completing a malaria RDT is error-prone, CHWs are often asked to carry completed RDTs back to their supervisors. Doing so makes RDTs susceptible to deterioration and introduces inefficiencies in the CHWs' workflow. In this work, we propose a smartphone-based RDT capture app, RDTScan, that facilitates the collection of high-quality RDT images to support CHWs in the field. RDTScan does not require an external adapter to control the image capture environment, but instead provides real-time guidance using image processing to obtain the best image possible. During our evaluation study, we found that RDTScan had 98.1% sensitivity and 99.7% specificity against visual inspection of the RDTs. RDTScan helped CHWs capture high-quality RDT images within 18 seconds while enabling a better RDT workflow.

Breathalyzers, the standard quantitative method for assessing inebriation, are primarily owned by law enforcement and used only after a potentially inebriated individual is caught driving. However, not everyone has access to such specialized hardware. We present drunk user interfaces: smartphone user interfaces that measure how alcohol affects a person's motor coordination and cognition using performance metrics and sensor data. We examine five drunk user interfaces and combine them to form the "DUI app". DUI uses machine learning models trained on human performance metrics and sensor data to estimate a person's blood alcohol level (BAL). We evaluated DUI on 14 individuals in a week-long longitudinal study wherein each participant used DUI at various BALs. We found that with a global model that accounts for user-specific learning, DUI can estimate a person's BAL with an absolute mean error of 0.005% ± 0.007% and a Pearson's correlation coefficient of 0.96 with breathalyzer measurements.

Smartphones and tablets are often used in dynamic environments that force users to break focus and attend to their surroundings, creating a form of "situational impairment." Current mobile devices have no ability to sense when users divert or restore their attention, let alone provide support for resuming tasks. We therefore introduce SwitchBack, a system that allows mobile device users to resume tasks more efficiently. SwitchBack is built upon Focus and Saccade Tracking (FAST), which uses the front-facing camera to determine when the user is looking and how their eyes are moving across the screen. In a controlled study, we found that FAST can identify how many lines the user has read in a body of text within a mean absolute percent error of just 3.9%. We then tested SwitchBack in a dual focus-of-attention task, finding that SwitchBack improved average reading speed by 7.7% in the presence of distractions.

Emerging uses of imaging technology for consumers cover a wide range of application areas from health to interaction techniques; however, typical cameras primarily transduce light from the visible spectrum into only three overlapping components of the spectrum: red, blue, and green. In contrast, hyperspectral imaging breaks down the electromagnetic spectrum into more narrow components and expands coverage beyond the visible spectrum. While hyperspectral imaging has proven useful as an industrial technology, its use as a sensing approach has been fragmented and largely neglected by the UbiComp community. We explore an approach to make hyperspectral imaging easier and bring it closer to the end-users. HyperCam provides a low-cost implementation of a multispectral camera and a software approach that automatically analyzes the scene and provides a user with an optimal set of images that try to capture the salient information of the scene. We present a number of use-cases that demonstrate HyperCam's usefulness and effectiveness.

The different electronic devices we use on a daily basis produce distinct electromagnetic radiation due to differences in their underlying electrical components. We present MagnifiSense, a low-power wearable system that uses three passive magneto-inductive sensors and a minimal ADC setup to identify the device a person is operating. MagnifiSense achieves this by analyzing near-field electromagnetic radiation from common components such as the motors, rectifiers, and modulators. We conducted a staged, in-the-wild evaluation where an instrumented participant used a set of devices in a variety of settings in the home such as cooking and outdoors such as commuting in a vehicle. MagnifiSense achieves a classification accuracy of 82.6% using a model-agnostic classifier and 94.0% using a model-specific classifier. In a 24-hour naturalistic deployment, MagnifiSense correctly identified 25 of the total 29 events, while achieving a low false positive rate of 0.65% during 20.5 hours of non-activity.

This paper presents SAIL, a Single Access Point Based Indoor Localization system. Although there have been advances in WiFi-based positioning techniques, we find that existing solutions either require a dense deployment of access points (APs), manual fingerprinting, energy hungry WiFi scanning, or sophisticated AP hardware. We design SAIL using a single commodity WiFi AP to avoid these restrictions. SAIL computes the distance between the client and an AP using the propagation delay of the signal traversing between the two, combines the distance with smartphone dead-reckoning techniques, and employs geometric methods to ultimately yield the client's location using a single AP. SAIL combines physical layer (PHY) information and human motion to compute the propagation delay of the direct path by itself, eliminating the adverse effect of multipath and yielding sub-meter distance estimation accuracy. Furthermore, SAIL systematically addresses some of the common challenges towards dead-reckoning using smartphone sensors and achieves 2-5x accuracy improvements over existing techniques. We have implemented SAIL on commodity wireless APs and smartphones. Evaluation in a large-scale enterprise environment with 10 mobile users demonstrates that SAIL can capture the user's location with a mean error of 2.3m using just a single AP.

This video presents a demo of indoor localization in multiple settings. In the demo, a user walks with a smartphone and the user's location is shown on the phone's screen in real time. Our system, called Unsupervised Indoor Localization (UnLoc) utilizes the sensor data from smartphones to learn "invisible landmarks" in the environment. Example landmarks could be a unique magnetic fluctuation experienced when the phone is near a water-cooler, or a distinct gyroscope rotation when the user turns a corner. We use these indoor "landmarks" to periodically reset the user's location. To track the user between these landmarks, we use an optimized variant of dead reckoning, ultimately leading to a robust location tracking system. We call our system UnLoc, since the landmarks are generated in an unsupervised manner, requiring no manual effort or floorplan of the building. The demo describes the high level intuitions, shows UnLoc in operation, and shares experiences from running UnLoc in various real-world environments.

We propose UnLoc [1], an unsupervised indoor localization scheme that bypasses the need for war-driving. Our key observation is that certain locations in an indoor environment present an identifiable signature on one or more sensing dimensions. An elevator, for instance, imposes a distinct pattern on a smartphone's accelerometer; a specific spot may experience an unusual magnetic fluctuation. This form of urban sensing and activity recognition has already been demonstrated in literature [2, 3], but not yet applied in pure localization applications. We hypothesize that these kind of signatures naturally exist in the environment and can be envisioned as internal landmarks of a building. Mobile devices that "sense" these landmarks can recalibrate their locations, while dead-reckoning schemes can track them between landmarks. Neither war-driving nor floorplans are necessary - the system simultaneously computes the locations of users and landmarks, in a manner so that they converge reasonably quickly. We believe this is an unconventional approach to indoor localization, holding promise for real-world deployment.