Tracking for people living with dementia and their care partners is primarily focused on quantified dementia symptoms presented to care partners. However, what people living with dementia want to track, what other aspects of dementia care partners wish to know, and how tracking fits within the care relationship remain to be identified. We performed an exploratory study in which eight people living with dementia and nine care partners provided iterative design feedback on a system concept: one that captures conversational data from a robot and visualizes it through a speech-tracking mobile application. Through reflexive thematic analysis, we found that people living with dementia wanted to use the system to maintain autonomy, especially by talking about their symptoms with the robot and using tracked information as a memory aid. Care partners valued numerical insights into the cognitive progress of people living with dementia only when accompanied by clear calls to action that supported them in their caregiver roles. Simultaneously, in their relational roles as spouses or children, care partners valued tracking memories and discussion points to understand their loved ones better. Our results suggest that providing related but distinct information tailored to each user’s needs can support both in their care relationship.

There is great interest in applying artificial intelligence (AI) techniques to healthcare issues such as Autism, particularly in combination with digital health technologies (robots, wearables, smartphones, etc.) in user homes. However, a critical challenge is that modern AI techniques like deep learning (DL) typically require large datasets with millions of samples, yet in healthcare we are often working with smaller clinical samples (<50 participants). To address that challenge, we need to develop new approaches that can learn more efficiently from less data. In this paper, we propose a novel approach to few-shot learning (FSL) called SMOTE_FSL, which is applicable to traditional machine learning (ML) models, allowing them to work with smaller sample sizes as well as various types of healthcare data (not only image or text data). We compare SMOTE_FSL on two healthcare sensor datasets gathered using robots and wearables, with results showing SMOTE_FSL performs comparably to state-of-the-art DL-based FSL methods (e.g. autoencoders, generative adversarial networks [GAN]). That indicates such an approach holds potential to expand utilization of FSL to a broad range of healthcare data derived from smaller clinical sample sizes.

Socially assistive robots can be used as therapeutic technologies to address depression symptoms. Through three sets of workshops with individuals living with depression and clinicians, we developed design guidelines for a personalized therapeutic robot for adults living with depression. Building on the design of Therabot, workshop participants discussed various aspects of the robot's design, sensors, behaviors, and a robot connected mobile phone app. Similarities among participants and workshops included a preference for a soft textured exterior and natural colors and sounds. There were also differences - clinicians wanted the robot to be able to call for aid, while participants with depression differed in their degree of comfort in sharing data collected by the robot with clinicians.

Using socially assistive robots (SARs) as specialized companions for those living with depression to manage symptoms provides a unique opportunity for exploration of robotic systems as comfort objects. Moreover, the robotic components allow for specialized behavioral responses to particular stimuli, as preferred by the user. We have conducted semi-structured interviews with 10 participants about the zoomorphic robot’s Therabot™ desired behaviors and focus groups with five additional participants regarding the preferred sensors within the Therabot™ system. In this paper, using the data from interviews and focus groups, we explore SAR input and output for depression management. While participants overall expected the robot to respond in much similar ways as a well-trained service animal, they expressed interest in the robot understanding unique information about the environment and the user, such as when the user might need interaction.

A major research question in psycholinguistics relates to the phenomenon of linguistic relativity, which contends that the language one speaks influences how one thinks. Of particular interest is whether bilingual speakers shift cognitive paradigms when speaking different languages. Here, we conducted a human-agent interaction (HAI) study using a bilingual virtual avatar capable of autonomous speech during cooperative gameplay in two languages (Korean and English). We ran 40 participants, including 20 monolingual speakers (10 Korean, 10 English) and 20 Korean/English bilingual speakers, engaging the avatar during 30-minute game sessions. Comparison of speech patterns showed that bilingual speakers exhibited notable “cognitive shifts” in both languages while interacting with the avatar, which were markedly different from their monolingual counterparts. Interestingly, the virtual avatar’s own speech behavior also significantly changed during interaction with bilingual speakers, despite identical programming. As evidenced here, such cognitive shifts appear to impact the way humans interact with artificial agents.

Predicting when it is an artificial agent’s turn to speak/act during human-agent interaction (HAI) poses a significant challenge due to the necessity of real-time processing, context sensitivity, capturing complex human behavior, effectively integrating multiple modalities, and addressing class imbalance. In this paper, we present a novel deep learning network-based approach for predicting turn-taking events in HAI that leverages information from multiple modalities, including text, audio, vision, and context data. Our study demonstrates that incorporating additional modalities, including in-game context data, enables a more comprehensive understanding of interaction dynamics leading to enhanced prediction accuracy for the artificial agent. The efficiency of the model also permits potential real-time applications. We evaluated our proposed model on an imbalanced dataset of both successful and failed turn-taking attempts during an HAI cooperative gameplay scenario, comprising over 125,000 instances, and employed a focal loss function to address class imbalance. Our model outperformed baseline models, such as Early Fusion LSTM (EF-LSTM), Late Fusion LSTM (LF-LSTM), and the state-of-the-art Multimodal Transformer (Mult). Additionally, we conducted an ablation study to investigate the contributions of individual modality components within our model, revealing the significant role of speech content cues. In conclusion, our proposed approach demonstrates considerable potential in predicting turn-taking events within HAI, providing a foundation for future research with physical robots during human-robot interaction (HRI).

Socially-assistive robots (SARs) hold significant potential to transform the management of chronic healthcare conditions (e.g. diabetes, Alzheimer’s, dementia) outside the clinic walls. However doing so entails embedding such autonomous robots into people’s daily lives and home living environments, which are deeply shaped by the cultural and geographic locations within which they are situated. That begs the question whether we can design autonomous interactive behaviors between SARs and humans based on universal machine learning (ML) and deep learning (DL) models of robotic sensor data that would work across such diverse environments? To investigate this, we conducted a long-term user study with 26 participants across two diverse locations (United States and South Korea) with SARs deployed in each user’s home for several weeks. We collected robotic sensor data every second of every day, combined with sophisticated ecological momentary assessment (EMA) sampling techniques, to generate a large-scale dataset of over 270 million data points representing 173 hours of randomly-sampled naturalistic interaction data between the human and SAR. Models built on that data were capable of achieving nearly 84% accuracy for detecting specific interaction modalities (AUC 0.885) when trained/tested on the same location, though suffered significant performance drops when applied to a different location. Further analysis and participant interviews showed that was likely due to differences in home living environments in the US and Korea. The results suggest that our ability to create adaptable behaviors for robotic pets may be dependent on the human-robot interaction (HRI) data available for modeling.

Modeling smartphone keyboard dynamics as the foundation of an early warning system (EWS) for mood instability holds potential to expand the reach of healthcare beyond the traditional clinic wall’s, which may lead to better ongoing care for chronic mental illnesses such as bipolar disorder. Here, we investigate the feasibility of such a system using a real-world open-science dataset. In particular, we are interested in whether passive technology interaction patterns in real-world datasets reflect findings from more controlled research trials, and the implications for clinical care. Data from 328 people who downloaded an open-science app was analyzed using a variety of machine learning methods, including different modeling methods (random forests, gradient boosting, neural networks), different types of class rebalancing, and pre-processing techniques. The aim was to predict fluctuations in PHQ scores in the weeks before the fluctuation occurred. Various feature selection methods were also employed to identify the top features driving the predictive patterns (out of total 54 starting features). Results showed predictive accuracy around ∼90%, similar to controlled research trials, while revealing a number of interesting features (e.g. PTSD and mood instability) that suggest future research avenues. The findings from our analysis appear to indicate that real-world interaction data from smartphones can be utilized as an EWS monitoring tool for mood disorders like bipolar. We also discuss the broader applicability of ecological momentary assessment (EMA) approaches to connected systems combining different forms of pervasive technology interaction (smartphones, wearables, social robots) to track everyday health status.

Socially-Assistive Robots (SARs) hold great potential to revolutionize the way we manage chronic illness outside clinical settings, but a current limitation to their broad adoption for this purpose is the lack of "ground truth" around interactions between robots and humans in in-home settings.Such ground truth is a necessity for using robotic sensor data for machine learning models of patient activity patterns or to create AI to customize robotic interactive behavior autonomously.Traditional subjective recall-based data collection methods lack the fine-grained temporal detail to support such AI development, as well as suffering from "recall bias" effects.One potential solution to this challenge is to adapt novel forms of interaction assessment, such as ecological momentary assessment (EMA), to collect patient interaction data in real-time.Here we describe a pilot study utilizing such an EMA system with SARs.We describe the development of the EMA framework, theoretical design issues, and lessons learned.Preliminary machine learning results indicate 75-80% accuracy for detecting specific interaction modalities.We also discuss the potential utility of EMA for exploring cross-cultural differences with in-the-wild robot use, and as a tool to support participatory design research on robotics in healthcare settings.

A critical issue during human-agent and human-robot interaction is eliciting an intentional stance in the human interactor, whereas the human perceives the agent as a fully "intelligent" being with full agency towards their own intentions and desires. Eliciting such a stance, however, has proven elusive, despite work in cognitive science, robotics, and human-computer interaction over the past half-century. Here, we argue for a paradigm shift in our approach to this problem, based on a synthesis of recent evidence from social robotics and digital avatars. In short, in order to trigger an intentional stance in humans, perhaps our artificial agents need to adopt one about themselves.

OBJECTIVE: Ubiquitous technologies can be leveraged to construct ecologically relevant metrics that complement traditional psychological assessments. This study aims to determine the feasibility of smartphone-derived real-world keyboard metadata to serve as digital biomarkers of mood. MATERIALS AND METHODS: BiAffect, a real-world observation study based on a freely available iPhone app, allowed the unobtrusive collection of typing metadata through a custom virtual keyboard that replaces the default keyboard. User demographics and self-reports for depression severity (Patient Health Questionnaire-8) were also collected. Using >14 million keypresses from 250 users who reported demographic information and a subset of 147 users who additionally completed at least 1 Patient Health Questionnaire, we employed hierarchical growth curve mixed-effects models to capture the effects of mood, demographics, and time of day on keyboard metadata. RESULTS: We analyzed 86 541 typing sessions associated with a total of 543 Patient Health Questionnaires. Results showed that more severe depression relates to more variable typing speed (P < .001), shorter session duration (P < .001), and lower accuracy (P < .05). Additionally, typing speed and variability exhibit a diurnal pattern, being fastest and least variable at midday. Older users exhibit slower and more variable typing, as well as more pronounced slowing in the evening. The effects of aging and time of day did not impact the relationship of mood to typing variables and were recapitulated in the 250-user group. CONCLUSIONS: Keystroke dynamics, unobtrusively collected in the real world, are significantly associated with mood despite diurnal patterns and effects of age, and thus could serve as a foundation for constructing digital biomarkers.

Over the past two decades, data science or data analytics degree programs have begun to emerge, reflecting the world's demand for data specialists to make sense of the vast amounts of collected data in the sciences, engineering, business, and other domains. As degree creation has occurred mainly due to demand, ACM and other professional bodies have recently stepped in to provide curricular guidance. However, no \em shared global framework for data science as an academic discipline exists, making growth unfocused and driven by employer demands. More recently, the growth of artificial intelligence has also impacted data science programs. This working group builds on prior efforts and participant experiences to develop a global taxonomy of approaches to data science education and expectations for graduates of data science programs to \em think like data scientists.

This paper presents the results of research aimed at developing a methodology for the participatory design of social robots, which are meant to be incorporated into various social contexts (e.g. home, work) and establish social relations with people. In contrast to the dominant technologically driven robot development process, we aim to develop a socially robust and responsible approach to robot design using Participatory Design (PD) methods. The PD process builds on participants' self-identified issues and concerns, and develops robot concepts according to participants' interpretations of the capabilities and potential applications of robotic technologies. We present methodological insights from an ongoing PD project aimed at designing socially assistive robots with older adults diagnosed with depression and their therapists, and identify remaining challenges in this project. We particularly focus on supporting mutual learning between researchers and participants and on promoting active participation of older adults as "designers" (rather than consumers) as foundational aspects of PD. We conclude with reflections regarding how this work can contribute to the further development of social robots and relevant PD methodologies.

We performed an experimental study (n=48) of the effects of context congruency on human perceptions of robotic facial expressions across cultures (Western and East Asian individuals). We found that context congruency had a significant effect on human perceptions, and that this effect varied by the emotional valence of the context and facial expression. Moreover, these effects occurred regardless of the cultural background of the participants. In short, there were predictable patterns in the effects of congruent/incongruent environmental context on perceptions of robot affect across Western and East Asian individuals. We argue that these findings fit with a dynamical systems view of social cognition as an emergent phenomenon. Taking advantage of such context effects may ease the constraints for developing culturally-specific affective cues in human-robot interaction, opening the possibility to create culture-neutral models of robots and affective interaction.