Machine Learning (ML) algorithms play an increasingly critical role in remote health monitoring, automating health assessments, and extending medical professionals’ reach in caring for an aging population. However, ML models often provide predictions without quantifying uncertainty, which is essential for safe deployment in healthcare. This article presents a novel method referred to as Uncertainty Regions via Importance-Weighted Calibration (URIC) for automating the uncertainty-based prediction of multiple clinical cognitive health measures from continuous smartwatch sensor data. URIC leverages Conformal Prediction (CP), a rigorous Uncertainty Quantification (UQ) framework that guarantees user-defined coverage (e.g., ground truth is in the predicted region with 95% probability). The key innovation of URIC is constructing smaller, more interpretable prediction regions by assigning importance weights to calibration examples. It forms prediction regions by using the most relevant calibration examples, ensuring tight regions without sacrificing coverage. We evaluate URIC for predicting the cognitive health measures of 157 adults who were either cognitively healthy or experienced Mild Cognitive Impairment (MCI). Compared to other UQ methods, our method constructs smaller prediction regions across all multi-target combinations while maintaining the user-specified coverage. This approach can be integrated into human-ML collaborative systems to improve diagnostic accuracy and interpretability in sensitive multi-target healthcare tasks.

Forecasting future health status is beneficial for understanding health patterns and providing anticipatory support for cognitive and physical health difficulties. In recent years, generative Large Language Models (LLMs) have shown promise as forecasters. Though not traditionally considered strong candidates for numeric tasks, LLMs demonstrate emerging abilities to address various forecasting problems. They also provide the ability to incorporate unstructured information and explain their reasoning process. In this article, we explore whether LLMs can effectively forecast future self-reported health state. To do this, we utilized in-the-moment assessments of mental sharpness, fatigue, and stress from multiple studies, utilizing daily responses ( N = 106 participants) and responses that are accompanied by text descriptions of activities ( N = 32 participants). With these data, we constructed prompt/response pairs to predict a participant’s next answer. We fine-tuned several LLMs and applied chain-of-thought prompting evaluating forecasting accuracy and prediction explainability. Notably, we found that LLMs achieved the lowest Mean Absolute Error (MAE) overall (0.851), while gradient boosting achieved the lowest overall RMSE (1.356). When additional text context was provided, LLM forecasts achieved the lowest MAE for predicting mental sharpness (0.862), fatigue (1.000), and stress (0.414). These multimodal LLMs further outperformed the numeric baselines in terms of RMSE when predicting stress (0.947), although numeric algorithms achieved the best RMSE results for mental sharpness (1.246) and fatigue (1.587). This study offers valuable insights for future applications of LLMs in health-based forecasting. The findings suggest that LLMs, when supplemented with additional text information, can be effective tools for improving health forecasting accuracy.

In this study, we introduce and validate a computational method to detect lifestyle change that occurs in response to a multi-domain healthy brain aging intervention. To detect behavior change, digital behavior markers (DM) are extracted from smartwatch sensor data and a Permutation-based Change Detection (PCD) algorithm quantifies the change in marker-based behavior from a pre-intervention, one-week baseline. To validate the method, we verify that changes are successfully detected from synthetic data with known pattern differences. Next, we employ this method to detect overall behavior change for n=28 BHI subjects and n=17 age-matched control subjects. For these individuals, we observe a monotonic increase in behavior change from the baseline week with a slope of 0.7460 for the intervention group and a slope of 0.0230 for the control group. Finally, we utilize a random forest algorithm to perform leave-one-subject-out prediction of intervention versus control subjects based on digital marker delta values. The random forest predicts whether the subject is in the intervention or control group with an accuracy of 0.87. This work has implications for capturing objective, continuous data to inform our understanding of intervention adoption and impact.

New modes of technology are offering unprecedented opportunities to unobtrusively collect data about people's behavior. While there are many use cases for such information, we explore its utility for predicting multiple clinical assessment scores. Because clinical assessments are typically used as screening tools for impairment and disease, such as mild cognitive impairment (MCI), automatically mapping behavioral data to assessment scores can help detect changes in health and behavior across time. In this article, we aim to extract behavior markers from two modalities, a smart home environment and a custom digital memory notebook app, for mapping to 10 clinical assessments that are relevant for monitoring MCI onset and changes in cognitive health. Smart-home-based behavior markers reflect hourly, daily, and weekly activity patterns, while app-based behavior markers reflect app usage and writing content/style derived from free-form journal entries. We describe machine learning techniques for fusing these multimodal behavior markers and utilizing joint prediction. We evaluate our approach using three regression algorithms and data from 14 participants with MCI living in a smart-home environment. We observed moderate to large correlations between predicted and ground-truth assessment scores, ranging from r = 0.601 to r = 0.871 for each clinical assessment.

This article introduces RAS, a cyber-physical system that supports individuals with memory limitations to perform daily activities in their own homes. RAS represents a partnership between a smart home, a robot, and software agents. When smart home residents perform activities, RAS senses their movement in the space and identifies the current activity. RAS tracks activity steps to detect omission errors. When an error is detected, the RAS robot finds and approaches the human with an offer of assistance. Assistance consists of playing a video recording of the entire activity, showing the omitted activity step, or guiding the resident to the object that is required for the current step. We evaluated RAS performance for 54 participants performing three scripted activities in a smart home testbed and for 2 participants using the system over multiple days in their own homes. In the testbed experiment, activity errors were detected with a sensitivity of 0.955 and specificity of 0.992. RAS assistance was performed successfully with a rate of 0.600. In the in-home experiments, activity errors were detected with a combined sensitivity of 0.905 and a combined specificity of 0.988. RAS assistance was performed successfully for the in-home experiments with a rate of 0.830.

Ecological Momentary Assessment (EMA) is an in-the-moment data collection method which avoids retrospective biases and maximizes ecological validity. A challenge in designing EMA systems is finding a time to ask EMA questions that increases participant engagement and improves the quality of data collection. In this work, we introduce SEP-EMA, a machine learning-based method for providing transition-based context-aware EMA prompt timings. We compare our proposed technique with traditional time-based prompting for 19 individuals living in smart homes. Results reveal that SEP-EMA increased participant response rate by 7.19% compared to time-based prompting. Our findings suggest that prompting during activity transitions makes the EMA process more usable and effective by increasing EMA response rates and mitigating loss of data due to low response rates.

In order to meet the health needs of the coming "age wave," technology needs to be designed that supports remote health monitoring and assessment. In this study we design clinician in the loop (CIL), a clinician-in-the-loop visual interface, that provides clinicians with patient behavior patterns, derived from smart home data. A total of 60 experienced nurses participated in an iterative design of an interactive graphical interface for remote behavior monitoring. Results of the study indicate that usability of the system improves over multiple iterations of participatory design. In addition, the resulting interface is useful for identifying behavior patterns that are indicative of chronic health conditions and unexpected health events. This technology offers the potential to support self-management and chronic conditions, even for individuals living in remote locations.

As members of an increasingly aging society, one of our major priorities is to develop tools to detect the earliest stage of age-related disorders such as Alzheimer's Disease (AD). The goal of this paper is to evaluate the possibility of using unobtrusively collected activity-aware smart home behavior data to detect the multimodal symptoms that are often found to be impaired in AD. After gathering longitudinal smart home data for 29 older adults over an average duration of 2 years, we automatically labeled the data with corresponding activity classes and extracted time-series statistics containing ten behavioral features. Mobility, cognition, and mood were evaluated every six months. Using these data, we created regression models to predict symptoms as measured by the tests and a feature selection analysis was performed. Classification models were built to detect reliable absolute changes in the scores predicting symptoms and SmoteBOOST and wRACOG algorithms were used to overcome class imbalance where needed. Results show that all mobility, cognition, and depression symptoms can be predicted from activity-aware smart home data. Similarly, these data can be effectively used to predict reliable changes in mobility and memory skills. Results also suggest that not all behavioral features contribute equally to the prediction of every symptom. Future work therefore can improve model sensitivity by including additional longitudinal data and by further improving strategies to extract relevant features and address class imbalance. The results presented herein contribute toward the development of an early change detection system based on smart home technology.

Smart home environments offer an unprecedented opportunity to unobtrusively monitor human behavior. Sensor data collected from smart homes can be labeled using activity recognition to help determine whether relationships exist between behavior in the home and health changes. To detect and analyze behavior changes that accompany health events, we introduce the behavior change detection (BCD) approach. BCD detects activity timing and duration changes between windows of time, determines the significance of the detected changes, and analyzes the nature of the changes. We demonstrate our approach using two case studies for older adults living in smart homes who experienced major health events, including cancer treatment and insomnia. Our algorithm detects behavior changes consistent with the medical literature for these cases. The results suggest the changes can be automatically detected using BCD. The proposed smart home, activity recognition algorithms, and change detection approach are useful data mining techniques for understanding the behavioral effects of major health conditions.

Smart home technologies offer potential benefits for assisting clinicians by automating health monitoring and well-being assessment. In this paper, we examine the actual benefits of smart home-based analysis by monitoring daily behavior in the home and predicting clinical scores of the residents. To accomplish this goal, we propose a clinical assessment using activity behavior (CAAB) approach to model a smart home resident's daily behavior and predict the corresponding clinical scores. CAAB uses statistical features that describe characteristics of a resident's daily activity performance to train machine learning algorithms that predict the clinical scores. We evaluate the performance of CAAB utilizing smart home sensor data collected from 18 smart homes over two years. We obtain a statistically significant correlation ( r=0.72) between CAAB-predicted and clinician-provided cognitive scores and a statistically significant correlation ( r=0.45) between CAAB-predicted and clinician-provided mobility scores. These prediction results suggest that it is feasible to predict clinical scores using smart home sensor data and learning-based data analysis.

Individuals with cognitive impairment can benefit from intervention strategies like recording important information in a memory notebook. However, training individuals to use the notebook on a regular basis requires a constant delivery of reminders. In this work, we design and evaluate machine learning-based methods for providing automated reminders using a digital memory notebook interface. Specifically, we identify transition periods between activities as times to issue prompts. We consider the problem of detecting activity transitions using supervised and unsupervised machine learning techniques, and find that both techniques show promising results for detecting transition periods. We test the techniques in a scripted setting with 15 individuals. Motion sensors data is recorded and annotated as participants perform a fixed set of activities. We also test the techniques in an unscripted setting with 8 individuals. Motion sensor data is recorded as participants go about their normal daily routine. In both the scripted and unscripted settings a true positive rate of greater than 80% can be achieved while maintaining a false positive rate of less than 15%. On average, this leads to transitions being detected within 1 minute of a true transition for the scripted data and within 2 minutes of a true transition on the unscripted data.

One of the many services that intelligent systems can provide is the ability to analyze the impact of different medical conditions on daily behavior. In this study, we use smart home and wearable sensors to collect data, while ( n = 84) older adults perform complex activities of daily living. We analyze the data using machine learning techniques and reveal that differences between healthy older adults and adults with Parkinson disease not only exist in their activity patterns, but that these differences can be automatically recognized. Our machine learning classifiers reach an accuracy of 0.97 with an area under the ROC curve value of 0.97 in distinguishing these groups. Our permutation-based testing confirms that the sensor-based differences between these groups are statistically significant.

Individuals with cognitive impairment have difficulty successfully performing activities of daily living, which can lead to decreased independence. In order to help these individuals age in place and decrease caregiver burden, technologies for assistive living have gained popularity over the last decade. In this work, a context-aware prompting system is implemented, augmented by a smart phone to determine prompt situations in a smart home environment. While context-aware systems use temporal and environmental information to determine context, we additionally use ambulatory information from accelerometer data of a phone which also acts as a mobile prompting device. A pilot study with healthy young adults is conducted to examine the feasibility of using a smart phone interface for prompt delivery during activity completion in a smart home environment.

Individuals with cognitive impairment have difficulty successfully performing activities of daily living, which can lead to decreased independence. In order to help these individuals age in place and decrease caregiver burden, technologies for assistive living have gained popularity over the last decade. This demo illustrates the implementation of a context-aware prompting system augmented by a smart phone to determine prompt situations in a smart home environment. While context-aware systems use temporal and environmental information to determine context, we additionally use ambulatory information from accelerometer data of a phone which also acts as a mobile prompting device.

The machine learning and pervasive sensing technologies found in smart homes offer unprecedented opportunities for providing health monitoring and assistance to individuals experiencing difficulties living independently at home. In order to monitor the functional health of smart home residents, we need to design technologies that recognize and track activities that people normally perform as part of their daily routines. Although approaches do exist for recognizing activities, the approaches are applied to activities that have been pre-selected and for which labeled training data is available. In contrast, we introduce an automated approach to activity tracking that identifies frequent activities that naturally occur in an individual's routine. With this capability we can then track the occurrence of regular activities to monitor functional health and to detect changes in an individual's patterns and lifestyle. In this paper we describe our activity mining and tracking approach and validate our algorithms on data collected in physical smart environments.