Foundation models tested for clinical practice using human-designed metrics may mask fundamental differences in information processing. We investigated this using the clock drawing test (CDT), a cognitive screening tool. Three foundation models achieved 94% accuracy on conventional metrics, matching experts. However, upon decomposing the CDT into 24 questions across five cognitive domains, results diverged significantly. In cases with unanimous model agreement, they still disagreed with human raters in 22% cases. Performance varied drastically with 88% alignment with humans on rule-based executive questions but only 46% on context-dependent anticipatory thinking questions. We observed that models abstained three times more than humans, primarily owing to poor data quality. These findings show standard clinical evaluation metrics fail to capture how foundation models process information. High aggregate accuracy obscures component-level failures. We contribute a systematic evaluation of frontier models’ healthcare capabilities, demonstrate theory-driven task decomposition, and discuss design implications for better human-AI collaborative systems.

Self-experimentation, or using tracked data to systematically answer health and wellbeing questions via hypothesis testing, has significant potential to support personal health. However, technological support for self-experimentation has focused on expert-designed self-experiments for specific health conditions, limiting people’s ability to design their own rigorous experiments. To address this gap, we developed CASEbot (Conversation Agent for Self-Experimentation), an LLM-powered chatbot using a theory-driven approach to guide users through designing well-structured, personalized, and safe self-experiments. We conducted a within-subjects, mixed-methods study with 42 participants comparing CASEbot to a traditional worksheet-based approach. When formally comparing the experiment rigor and specificity, most participants designed better experiments using CASEbot. They appreciated CASEbot’s conversational approach, which prompted them to surface everyday constraints and proactively raised safety concerns, but some found the platform too rigid in its recommendations. We discuss opportunities for future generative AI self-experimentation systems for health to balance structured guidance with user autonomy.

AI chatbots are increasingly integrated into various sectors, including healthcare. We examine their role in responding to queries related to Alzheimer’s Disease and Related Dementias (AD/ADRD). We obtained real-world queries from AD/ADRD online communities (OCs)—Reddit (r/Alzheimers) and ALZConnected. First, we conducted a small-scale qualitative examination where we prompted ChatGPT, Bard, and Llama-2 with 101 OC posts to generate responses and compared them with OC responses through inductive coding and thematic analysis. We found that although AI can provide emotional and informational support like OCs, they do not engage in deeper conversations, provide references, and share personal experiences. These insights motivated us to conduct a large-scale quantitative examination of comparing AI (GPT) and OC responses (90K) to 13.5K posts, in terms of psycholinguistics, lexico-semantics, and content. AI responses tend to be more verbose, readable, and complex. AI responses exhibited greater empathy, but more formal and analytical language, lacking personal narratives and linguistic diversity. We found that various LLMs, including GPT, Llama, and Mistral, exhibit consistent patterns in responding to AD/ADRD-related queries, underscoring the robustness of our insights across LLMs. Our study sheds light on the potential of AI in digital health and underscores design considerations of AI to complement human interactions.

Chronic liver disease can lead to neurological conditions that result in coma or death. Although early detection can allow for intervention, testing is infrequent and unstandardized. Beacon is a device for at-home patient self-measurement of cognitive function via critical flicker frequency, which is the frequency at which a flickering light appears steady to an observer. This paper presents our efforts in iterating on Beacon's hardware and software to enable at-home use, then reports on an at-home deployment with 21 patients taking measurements over 6 weeks. We found that measurements were stable despite being taken at different times and in different environments. Finally, through interviews with 15 patients and 5 hepatologists, we report on participant experiences with Beacon, preferences around how CFF data should be presented, and the role of caregivers in helping patients manage their condition. Informed by our experiences with Beacon, we further discuss design implications for home health devices.

Collaborative care management is an evidence-based approach to integrated psychosocial care for patients with comorbid cancer and depression. Prior work highlights challenges in patient-provider collaboration in navigating parallel cancer care and psychosocial care journeys of these patients. We design and deploy SCOPE , a platform for technology-enhanced collaborative care combining a patient-facing mobile app with a provider-facing registry. We examine SCOPE through a total of 45 interviews with patients and providers conducted in SCOPE 's 15 months of design and development and 24 Months of SCOPE 's deployment for actual care in 6 cancer clinics. We find that: (1) SCOPE supported patient engagement in its underlying collaborative care and behavioral activation interventions, (2) patient-generated data in SCOPE improved patient-provider collaboration between and within in-person sessions, (3) SCOPE supported providers in delivering care and improved care team collaboration, (4) experience with SCOPE created evolving expectations for collaboration around data, and (5) SCOPE 's deployment in actual care surfaced important implementation barriers. We discuss the implications of our findings in terms of designing for engagement with behavioral health interventions, negotiating patient data sharing and provider responsiveness, supporting personalized self-tracking goals in evidence-based interventions, exploring the role of digital health navigators in technology-enhanced care, and the need for flexibility in aligning technology-supported interventions to patient needs.

Alzheimer's Disease and Related Dementias (AD/ADRD) are progressive neurodegenerative conditions that impair memory, thought processes, and functioning. Family caregivers of individuals with AD/ADRD face significant mental health challenges due to long-term caregiving responsibilities. Yet, current support systems often overlook the evolving nature of their mental wellbeing needs. Our study examines caregivers' mental wellbeing concerns, focusing on the practices they adopt to manage the burden of caregiving and the technologies they use for support. Through semi-structured interviews with 25 family caregivers of individuals with AD/ADRD, we identified the key causes and effects of mental health challenges and developed a temporal mapping of how caregivers' mental wellbeing evolves across three distinct stages of the caregiving journey. Additionally, our participants shared insights into improvements for existing mental health technologies, emphasizing the need for accessible, scalable, and personalized solutions that adapt to caregivers' changing needs over time. These findings offer a foundation for designing dynamic, stage-sensitive interventions that holistically support caregivers' mental wellbeing, benefiting both caregivers and care recipients.

EEG signals contain highly sensitive information about an individual's mental state, cognitive processes, and health conditions, making privacy preservation crucial. With the rise of commercial headwear capable of capturing EEG signals, developing robust mechanisms for ensuring privacy of such data is imperative. This work aims to protect EEG data privacy in cloud-based processing systems by sending intermediate output after neural network layer splitting to the cloud. We propose a novel holistic Combined Privacy Metric (CPM) that quantifies privacy leakage between raw EEG signals and intermediate outputs. Our study focuses on EEG-based seizure detection using a 1D CNN architecture, achieving accuracy of 96.25%. We evaluate various splitting configurations to optimize the trade-off between privacy preservation and computational efficiency. We find that splitting after the second convolutional layer achieves a CPM of 0.82 with a modest client-side model size of 509kB. This approach significantly enhances EEG data privacy while enabling effective cloud-based analysis, potentially facilitating wider adoption of secure EEG technologies in healthcare and research applications.

Self-tracked health data can help people and their health providers understand and manage chronic conditions. This paper examines personal informatics practices and challenges in migraine, a condition characterized by unpredictable, intermittent, and poorly-understood symptoms. To investigate how people with migraine track and use data related to their condition, we surveyed 279 people with migraine and conducted semi-structured interviews with 13 survey respondents and 6 health providers. We find four distinct goals people bring to tracking and data: 1) answering questions about migraines, 2) predicting and preventing migraines, 3) monitoring and managing migraines over time, and 4) enabling motivation and social recognition. Each goal suggests different needs for the design of tools to support migraine tracking. We also find needs resulting from an individual's goals evolving over time, their varied personal experiences, and their communication and collaboration with providers. We discuss these goals and needs in terms of opportunities for personal informatics tools to facilitate learning to: 1) avoid common pitfalls; 2) support customization and flexibility; 3) account for burden, negativity, and lapsing; and 4) support management with uncertainty.

Diagnostic self-tracking, the recording of personal information to diagnose or manage a health condition, is a common practice, especially for people with chronic conditions. Unfortunately, many who attempt diagnostic self-tracking have trouble accomplishing their goals. People often lack knowledge and skills needed to design and conduct scientifically rigorous experiments, and current tools provide little support. To address these shortcomings and explore opportunities for diagnostic self-tracking, we designed, developed, and evaluated a mobile app that applies a self-experimentation framework to support patients suffering from irritable bowel syndrome (IBS) in identifying their personal food triggers. TummyTrials aids a person in designing, executing, and analyzing self-experiments to evaluate whether a specific food triggers their symptoms. We examined the feasibility of this approach in a field study with 15 IBS patients, finding that participants could use the tool to reliably undergo a self-experiment. However, we also discovered an underlying tension between scientific validity and the lived experience of self-experimentation. We discuss challenges of applying clinical research methods in everyday life, motivating a need for the design of self-experimentation systems to balance rigor with the uncertainties of everyday life.

OBJECTIVE: To describe an interdisciplinary and methodological framework for applying single case study designs to self-experimentation in personalized health. The authors examine the framework's applicability to various health conditions and present an initial case study with irritable bowel syndrome (IBS). METHODS AND MATERIALS: An in-depth literature review was performed to develop the framework and to identify absolute and desired health condition requirements for the application of this framework. The authors developed mobile application prototypes, storyboards, and process flows of the framework using IBS as the case study. The authors conducted three focus groups and an online survey using a human-centered design approach for assessing the framework's feasibility. RESULTS: All 6 focus group participants had a positive view about our framework and volunteered to participate in future studies. Most stated they would trust the results because it was their own data being analyzed. They were most concerned about confounds, nonmeaningful measures, and erroneous assumptions on the timing of trigger effects. Survey respondents (N = 60) were more likely to be adherent to an 8- vs 12-day study length even if it meant lower confidence results. DISCUSSION: Implementation of the self-experimentation framework in a mobile application appears to be feasible for people with IBS. This framework can likely be applied to other health conditions. Considerations include the learning curve for teaching self-experimentation to non-experts and the challenges involved in operationalizing and customizing study designs. CONCLUSION: Using mobile technology to guide people through self-experimentation to investigate health questions is a feasible and promising approach to advancing personalized health.