Conversational agents (CAs) are transforming human-computer interaction, evolving from text-based chatbots to digital humans (DHs) capable of rich emotional expression. This study explores integrating neural and physiological signals into the perception module of CAs to enable real-time emotion detection and empathetic responses. We conducted a user study in which participants engaged with a DH about emotional topics. The DH mirrored participants’ emotions in real-time using neural and physiological cues. Results showed that users experienced stronger emotions and greater engagement during interactions with the Empathetic DH, highlighting the benefits of these signals for enhancing empathy. However, challenges remain, including recognition accuracy, emotional transition timing, individual differences, and limited voice modulation. Addressing these issues is key to advancing empathetic digital agents. This research demonstrates the promise of real-time physiological and neural emotion recognition for building emotionally intelligent CAs that foster deeper, more meaningful human-agent interactions.

Mild cognitive impairment (MCI) is an early stage of cognitive decline that significantly increases the risk of dementia, making early interventions crucial for maintaining cognitive health in older adults. Our research takes a co-design approach to understand, design, and evaluate how socially assistive robots and a virtual human can promote lifestyle changes for people with MCI (pwMCI), potentially improving their cognitive health. Through an iterative refinement process, we aim to develop adaptable, user-friendly, and sustainable technologies that foster long-term engagement across diverse settings. Ultimately, our goal is to improve cognitive health, quality of life, emotional well-being, and loneliness in pwMCI.

Research has shown that the composition of breath can differ based on the human's behavioral patterns and mental and physical states immediately before being collected. These breath-collection techniques have also been extended to observe the general processes occurring in groups of humans and can link them to what those groups are collectively experiencing. In this research, we applied machine learning techniques to the breath data collected from cinema audiences. These techniques included XGBOOST Regression, Hierarchical Clustering, and Item Basket analyses created using the Apriori algorithm. They were conducted to find associations between the biomarkers in the crowd's breath and the movie's audio-visual stimuli and thematic events. This analysis enabled us to directly link what the group was experiencing and their biological response to that experience. We first extracted visual and auditory features from a movie to achieve this. We compared it to the biomarkers in the crowd's breath using regression and pattern mining techniques. Our results supported the theory that a crowd's collective experience directly correlates to the biomarkers in the crowd's breath. Consequently, these findings suggest that visual and auditory experiences have predictable effects on the human body that can be monitored without requiring expensive or invasive neuroimaging techniques.

Clinical empathy has been associated with many positive outcomes, including patient trust and satisfaction. Physicians can demonstrate clinical empathy through verbal statements and non-verbal behaviors, such as head nodding. The use of verbal and non-verbal empathy behaviors by healthcare robots may also positively affect patient outcomes. The current study examined whether the use of robot verbal empathy statements and head nodding during a video recorded interaction between a healthcare robot and patient improved participant trust and satisfaction. One hundred participants took part in the experiment, online through Amazon Mechanical Turk. They were randoimnized to watch one of four videos depicting an interaction with a `patient' and a Nao robot that (1) either made empathetic or neutral statements, and (2) either nodded its head when listening to the patient or did not. Results showed that the use of empathetic statements by the healthcare robot significantly increased participant perceptions of robot empathy, trust and satisfaction, and reduced robot distrust. No significant findings were revealed in relation to robot head nodding. The positive effects of empathy statements support the model of Robot-Patient Communication, which theorizes that robot use of recommended clinical empathy behaviors can improve patient outcomes. The effects of healthcare robot nodding behavior needs to be further investigated.

No abstract available.

With the growing speed of automation, robots are taking on social care roles. In retirement villages and activity centers for older adults in New Zealand, the robotic seal Paro has become a valuable unpaid staff member contributing to the social life of people who struggle with the effects of dementia. This study aimed to investigate the role of Paro as an agent of care during a time with a steady push towards automation and digitalization of welfare services in New Zealand. As a part of the study four care workers, a family member of a former resident, and three researchers from aNew Zealand based robotics research group participated in in-depth interviews pertaining to the participant's own experiences, opinions, and motivations for using or working with social robots. Results found that Paro was used to increase residents' quality of life rather than reasons of automation for profit.

Digital humans are a highly realistic form of conversational computer agent. Eye gaze is a salient social cue that digital humans could use to facilitate rapport-building during conversations. However, eye gaze tendencies vary by gender and incorrect gaze patterns can have negative social implications. Analysis of observational data during human conversations can help inform the development of eye gaze models for digital humans. This study aimed to identify the eye gaze patterns of women dyads during a rapport-building conversation, and to evaluate the effect of different gaze patterns on rapport, trust, and psychological outcomes. 36 adult women (18 dyads) completed the Relationship Closeness Induction Task while wearing eye tracking glasses. Subjective rapport, trust, and psychological measures were collected. Gaze patterns of women were found to change as the conversation content became more intimate; specifically, gaze aversions for thinking (p=.042), turn-taking (p=.025), and intimacy modulation increased in duration (p=.012). Furthermore, gaze patterns were associated with perceptions of the conversation partner. Displaying fewer cognitive gaze aversions was associated with greater closeness (p=.029) and trust perceptions (p=.035). Longer periods of direct gaze while speaking was associated with greater rapport (p=.040). Results will inform the development of a humanlike gaze model for female digital humans during intimate conversations and may be applicable to social robots.

Emotions are complicated psycho-physiological processes that are related to numerous external and internal changes in the body. They play an essential role in human-human interaction and can be important for human-machine interfaces. Automatically recognizing emotions in conversation could be applied in many application domains like health-care, education, social interactions, entertainment, and more. Facial expressions, speech, and body gestures are primary cues that have been widely used for recognizing emotions in conversation. However, these cues can be ineffective as they cannot reveal underlying emotions when people involuntarily or deliberately conceal their emotions. Researchers have shown that analyzing brain activity and physiological signals can lead to more reliable emotion recognition since they generally cannot be controlled. However, these body responses in emotional situations have been rarely explored in interactive tasks like conversations. This paper explores and discusses the performance and challenges of using brain activity and other physiological signals in recognizing emotions in a face-to-face conversation. We present an experimental setup for stimulating spontaneous emotions using a face-to-face conversation and creating a dataset of the brain and physiological activity. We then describe our analysis strategies for recognizing emotions using Electroencephalography (EEG), Photoplethysmography (PPG), and Galvanic Skin Response (GSR) signals in subject-dependent and subject-independent approaches. Finally, we describe new directions for future research in conversational emotion recognition and the limitations and challenges of our approach.

Participatory design includes stakeholders in the development of products intended to solve real-life challenges. Involving end users in the design of robots is vital for developing effective, useful, acceptable and user-friendly products that meet expectations, needs, and preferences. This four-year international project developed and evaluated a home-based robot for mood stabilization and cognitive improvement in older adults with mild cognitive impairment and age-related health needs. The daily-care robot was developed in collaboration with experts, carers, relatives, and older adults, through six phases. Two phases were dedicated to cognitive stimulation games. This paper provides a summary of the participatory design and mixed-methods evaluation processes undertaken to develop, refine, and test the robot. The final robot and games were acceptable to older adults, and useful for delivering stimulating activities and providing reminders for medication, health and wellbeing checks. Personalization is required to optimize human-robot interaction, and imagery and speech should be consistent with local users. Functions should be personalizable to accommodate individual health needs and preferences. This project highlights the importance of participatory design and testing robotics in end-user environments, as technical issues associated with long-term use were uncovered. Recommendations for future development and the design of assistive health robots are made.

Cognitive stimulation games delivered on robots may be able to improve cognitive functioning and delay decline in older adults. However, little is known about older adults’ in-depth opinions of robot-delivered games, as current research primarily focuses on technical development and one-off use. This article explores the usability, acceptability, and perceptions of community-dwelling older adults towards cognitive games delivered on a robot that incorporated movable interactive blocks. Semi-structured interviews were conducted with participants at the end of a 12-week cognitive stimulation games intervention delivered entirely on robots. Participants were 10 older adults purposively sampled from two retirement villages. A framework analysis approach was used to code data to predefined themes related to technology acceptance (perceived benefits, satisfaction, and preference), and usability (effectiveness, efficiency, and satisfaction). Results indicated that cognitive games delivered on a robot may be a valuable addition to existing cognitive stimulation activities. The robot was considered easy to use and useful in improving cognitive functioning. Future developments should incorporate interactive gaming tools, the use of social anthropomorphic robots, contrasting colour schemes to accommodate macular degeneration, and cultural-specific imagery and language. This will help cater to the preferences and age-related health needs of older adults, to ultimately enhance usability and acceptability.

In this paper we present ZenG, a neurofeedback ARapplication concept based on Zen Gardening to fostercreativity, self-awareness, and relaxation through embodiedinteractions in a mixed reality environment. We developedan initial prototype which combined physiological sensingthrough EEG with AR visualisation on the Magic LeapDisplay. We evaluated the prototype through preliminaryuser testing with 12 adults. Results suggest users found theexperience to be enjoyable and relaxing, however theapplication could be improved by including more featuresand functionality. ZenG shows the potential for AR toprovide immersive and interactive environments that couldpromote creativity and relaxation, providing solid groundsfor further research.

Women are underrepresented in robotics, and this may be partly due to the educational emphasis on mechanical applications rather than social applications of robotics. This study aimed to investigate whether teaching robotics using social robots increased girls' engagement compared to using more mechanical vex robots. 20 girls were recruited from school robotics classes. They were taught 30 minutes of VEX robotics and 30 minutes of social robotics in a counter-balanced order. Engagement was measured using questionnaires and observations. Results showed that girls were significantly more engaged in the social robot classes than the vex robot classes. This pilot study suggests a possible way to encourage more girls to study robotics.

A robot cannot be warm and friendly - or can it? To explore whether a robot can be a medical receptionist, we developed a robotic system for interacting with patients at a doctor's clinic, including acting friendly. We designed the robot to interact naturally with patients at the start and finish of a clinic visit. We investigated people's perceptions to the robot in a wizard-of-Oz study, where the participants interacted with the robot over four interactions. 40 participants evaluated the robot. The results indicate the participants thought the robot could be a friendly receptionist, especially after repeated interactions with the robot. However, the participants mainly thought the robot was friendly in a “professional” way, rather than a personal friend.

This paper presents a new version of robot receptionist system for healthcare facility environment. Our HealthBots consists of three subsystems: a receptionist robot system, a nurse assistant robot system, and a medical server. Our first version of receptionist robot, interacts with human at hospital reception, gives instructions to human verbally, but cannot understand what human says, so it uses a touch screen to get the response from human. In this paper, we design a receptionist robot that recognizes human face as well as speech, which enhances verbal interaction skill of robot. In addition, we design a reaction generation engine to generate appropriate reactive motions and speech. Moreover, we study which social skills are important to a hospital receptionist robot to enhance social interaction, such as friendliness and attention. We implemented perception modules, decision-making modules, and reaction modules to our HealthBots architecture, and did two case studies to find essential social skills for hospital receptionist robots.

Safety and acceptability are critical issues when people are interacting with robots in healthcare. Traditional robot arms are hard and inflexible, and may cause harm to users on impact. Soft robotic arms may be safer and more acceptable in these situations. Similarly robot fingers made of soft materials may be more acceptable and safer than fingers made of hard materials. Robot designers need to know how best to design arms for healthcare scenarios. There is limited research on the acceptability of soft robotic arms and fingers to date. This study aimed to investigate people's reactions to the touch of soft robotic arms and fingers, compared to more traditional hard forms, and to human arms and fingers. A second aim was to investigate people's perceptions of the usefulness of the arms and fingers for healthcare tasks. Thirty five community participants were blindfolded and participated in touching tasks for: 3 arms (soft robot, hard robot, and human) and four fingers (soft robot, medium robot, hard robot, and human) in a randomised order. The soft arm was rated significantly more human-like but also more fragile and less reliable than the hard arm. Participants perceived the soft arm as good for intimate tasks like washing the body, but the hard arm was perceived as better for weight-bearing tasks. The soft finger was rated significantly more creepy, fragile and unreliable than the other fingers. The medium robot finger was rated the most human-like of the robot fingers and was the favourite robot finger. These findings suggest people perceive soft robots to be more fragile than hard robots and as more appropriate for personal tasks. Overly soft fingers may be too creepy to be acceptable.

Robots have been proposed to reduce the costs of the provision of healthcare in rural settings, but as yet little research has tested this. This study investigated the feasibility and cost-effectiveness of a robot measuring routine vital signs in a family medicine clinic in a rural setting. The length of patient consultations was compared before (N = 85 patients) and after a robot was deployed in the clinic (N = 48 patients). A Cafero touchscreen robot took the patient's vital signs prior to the consultation and transferred the results to the medical professional's computer. Time-savings were calculated in New Zealand dollar terms and compared to the costs of the robot and its maintenance. Results showed that consultation lengths were cut by 18% on average (3 minutes and 13 seconds). If 20% of the clinics' annual consultations were augmented with the robot this translates to a total annual savings of NZ$19075. The annual cost of the robot was calculated to be NZ$9400 overs 5 years. Present value calculations of Benefit Cost result in a Benefit Cost ratio of 2.3. These results support the cost-effectiveness of the robot in a rural medical clinic. Further research is needed to improve the services provided by the robot and test it in a larger trial.

We describe an engaging interactive robot and the workflow design for incorporating such service robots in health care. The research is analyzing the long term usability of automated medication support for older people as they interact with a Stationary Robotic Medication Management System (StRoMMS). It delivers timely instructions and automated guidance as people take their daily medications in their independent living quarters in a retirement village. A pilot user study evaluated the hypothesized technological requirements of the robotic system and the clinical workflow requirements in the healthcare context. The novel contributions are our interactive robot and the workflow design. Following a “system of systems” design approach we determined that robots cannot work in isolation in a complex operational space such as healthcare. The value of introducing interactive robots in healthcare can be realized when the robot has interfaces with the healthcare system, which enhance the overall outcome and experience of the patient. Our research will inform the research community of the importance of the confluence of people, workflows and tools while designing healthcare robotics technology.

This paper presents the first version of a mobile service robot designed for older people. Six service application modules were developed with the key objective being successful interaction between the robot and the older people. A series of trials were conducted in an independent living facility at a retirement village, with the participation of 32 residents and 21 staff. In this paper, challenges of deploying the robot and lessons learned are discussed. Results show that the robot could successfully interact with people and gain their acceptance.

This study investigated whether attitudes and emotions towards robots predicted acceptance of a healthcare robot in a retirement village population. Residents (n = 32) and staff (n = 21) at a retirement village interacted with a robot for approximately 30 minutes. Prior to meeting the robot, participants had their heart rate and blood pressure measured. The robot greeted the participants, assisted them in taking their vital signs, performed a hydration reminder, told a joke, played a music video, and asked some questions about falls and medication management. Participants were given two questionnaires; one before and one after interacting with the robot. Measures included in both questionnaires were the Robot Attitude Scale (RAS) and the Positive and Negative Affect Schedule (PANAS). After using the robot, participants rated the overall quality of the robot interaction. Both residents and staff reported more favourable attitudes (p <; .05) and decreases in negative affect (p <; .05) towards the robot after meeting it, compared with before meeting it. Pre-interaction emotions and robot attitudes, combined with post-interaction changes in emotions and robot attitudes, were highly predictive of participants' robot evaluations (R = .88, p <; .05). The results suggest both pre-interaction emotions and attitudes towards robots, as well as experience with the robot, are important areas to monitor and address in influencing acceptance of healthcare robots in retirement village residents and staff. The results support an active cognition model that incorporates a feedback loop based on re-evaluation after experience.

As the proportion of people in the older age groups grows, demands on care providers increase. The ability of robotic technology to meet these demands is limited by a lack of acceptance by older people. This study investigates which tasks staff and residents in a retirement village would like a robot to assist with, as well as their attitudes towards robots and preferences for their appearance. Findings show that residents are more positive about robots than staff, and participants prefer a silver robot of 1.25 m height, with wheels and a screen on the body. Residents would most like the robot to assist with detecting falls, turning on and off appliances, lifting, cleaning, medication reminding, making phone calls and monitoring location. Making robots that fit these preferences may increase the acceptance of robotic assistants by older people.

Human-robot interaction (HRI) and user acceptance become critical when service robots start to provide a variety of assistance to users on a personal level. Limited research to date has studied the influence of users' attributes (such as age and gender) on the acceptance of service robots and the implications for HRI design. This paper describes the development of a social interactive healthcare robot named Charles, capable of measuring blood pressure. Using blood pressure monitoring as the service scenario, a user study was conducted to investigate the differences between two age groups (40 to 65 years and over 65 years) in attitudes and reactions before and after their interactions with Charles. The results showed few differences between the two age groups. A significant gender effect was found, with males having a more positive attitude toward robots in healthcare than females. This study reveals the importance of considering gender issues in the design of healthcare robots for older people. Overall, the performance of the robot was rated high, however the participants expressed desires to have more interactiveness and a better voice from the robot. According to our sample, age need not be a barrier to users' acceptance of healthcare robots.

There has been little previous research assessing people’s emotions and cognitions in response to different types of robot behaviour. This study investigated how people think and feel during interactions with robots who behave either well or poorly. 45 participants interacted with a B21r robot in a basic task to lead the robot along a marked path. Each participant was randomly assigned to either the robot following well or the robot following poorly. The most frequently reported emotions were frustration, fear, and happiness, and people commonly reported thoughts about the robot and also about themselves in the interaction. People reported more positive emotions in response to the good robot. Independently of group assignment, positive emotions during the task were associated with more positive evaluations of the robot and negative emotions were associated with more negative evaluations. These results suggest that robot designers should seek to maximize people’s positive emotions and minimize negative emotions to maximise the quality of human-robot interactions. Results also suggest that recognising emotions may be difficult for robots, and direct questioning may be an easier strategy.