This demonstration presents a novel human-robot interaction scenario in which two expressive, cost-effective robots debate the scientifically disproven notion that “the Earth is flat.” The demonstration draws attention to the importance of scientific literacy for fostering sustainability awareness. Human participants can join the discussion, challenging misinformation and guiding the robots toward evidence-based reasoning. By combining social robotics with dynamic, adaptive dialogues, this platform highlights how robust public discourse—free from myths and misconceptions—is critical for informed environmental stewardship, ultimately inspiring communities to embrace sustainable practices grounded in sound science.

Sign language is an essential communication tool, however, it can be highly challenging for non-deaf students to learn. We propose a Pepper robot based sign language teaching assistant called SignPepper, with the ability to communicate in both spoken and sign language. Using Whisper speech to text and Llama 3.3, SignPepper can engage in two way spoken lessons, with the ability to physically demonstrate signs to students. Furthermore, using 3D convolutional neural networks trained on sign language recognition, SignPepper can watch, analyze and give personalized feedback on students attempts at performing newly learned signs in real-time; including hand-based error localization.

This paper presents a two-stage facial expression mimicking system using the EveR-4 H22 robot, designed to improve Human-Robot Interaction (HRI) by accurately replicating human emotions. The system follows a two-step process: blendshape extraction, followed by optimized mapping functions that translate human expressions into the robot’s parameters. The parameter training employed Gradient Descent with Regularization, using the Adam Optimizer for 1 million iterations on the custom-labeled data with different emotional categories such as Sad, Fearful, Anger and more. Experimental results show improvements in emotional accuracy, with significant training outcomes that reduced Regularized L2 loss by 1,182 times, capable of accurately mimicking unseen facial emotional expression of an unseen individual. It holds potential applications in domains such as healthcare and customer service, as well as automating generation of demographically broader spectrum of emotional expressions.

Isolated sign language recognition is a challenging task involving the learning of complex relationships between spatial and temporal features. Due to the high complexity and relatively small datasets available, state-of-the-art methods often adopt language modeling and convolutional neural network based multimodal designs, achieving high accuracy at the cost of significant architectural complexity. Conceptually simpler, transformers have gained widespread adoption in related computer vision tasks, outperforming 3D convolutional network competitors. However, due to a lack of training data, video transformers struggle with sign language recognition and have not demonstrated competitive accuracy compared to 3D convolutional neural network designs. We introduce DeepSign, a family of vision transformer based sign language recognition models with superior performance to 3D convolutional neural network designs. Through careful model ablation we select the UniFormerV2 and VideoMAE V2 architectures and perform mixture of dataset pretraining. Our strongest model DeepSign UniFormerV2-L achieves state-of-the-art on the WLASL100 and MSASL100 benchmarks, producing 92.64% and 94% top-1 accuracies respectively. Armed with VideoMAE V2’s powerful pretrained backbone, DeepSign ViT base offers greater efficiency for a small accuracy tradeoff. We hope DeepSign will help advance future sign language research by providing strong foundational models to kickstart experiments.

Sign languages are widely used forms of communication by the deaf and hearing impaired communities. Due to a lack of qualified teachers, robot sign language teaching systems have been proposed, aiming to aid in sign language education. In this paper, we conduct a study utilizing the SignPepper system that we developed; a humanoid Pepper robot based system with capabilities in sign demonstration, verbal communication, and sign language recognition. Specifically, SignPepper adopts a 3D convolutional neural network trained on 100 American sign language signs, Whisper for speech recognition, ChatGPT 4o for context phrasing and Pepper’s built-in text-to-speech functionality. Our study consisted of 33 participants split into two groups, 18 participants were taught sign language by SignPepper whilst the other 15 were shown the same signs as videos. The same sign recognition neural network is used for evaluating the recall accuracy of students in both groups. Survey results showed the SignPepper group had higher sign recall accuracy, greater interest in learning more sign language and greater engagement. However, comfort during the lesson and comfort towards robotic platforms as teaching platforms was lower than the video group. The SignPepper group also rated instruction clarity as slightly lower. Our results indicate that the physical dexterity limits of the Pepper robot platform are a major limitation; as performance of signs may not match human experts exactly. Student comfort is also an area which requires future improvements. Nevertheless, SignPepper demonstrates strong viability for the adoption of robotic sign language teaching systems.

Multiple modalities can boost accuracy in the difficult task of Sign Language Recognition (SLR), however, each modality does not necessarily contribute the same quality of information. Current multi-modal approaches assign the same importance weightings to each modality, or set weightings based on unproven heuristics. This paper takes a systematic approach to find the optimal weights by performing grid search. Firstly, we create a multi-modal version of the RGB only WLASL100 data with additional hand crop and skeletal pose modalities. Secondly, we create a 3D CNN based weighted multi-modal sign language network (WMSLRnet). Finally, we run various grid searches to find the optimal weightings for each modality. We show that very minor adjustments in the weightings can have major effects on the final SLR accuracy. On WLASL100, we significantly outperform previous networks of similar design, and achieve high accuracy in SLR without highly complex pre-training schemes or extra data.

Aotearoa New Zealand has a strong and growing apple industry but struggles to access workers to complete skilled, seasonal tasks such as thinning. To ensure effective thinning and make informed decisions on a per-tree basis, it is crucial to accurately measure the crop load of individual apple trees. However, this task poses challenges due to the dense foliage that hides the fruitlets within the tree structure. In this paper, we introduce the vision system of an automated apple fruitlet thinning robot, developed to tackle the labor shortage issue. This paper presents the initial design, implementation, and evaluation specifics of the system. The platform straddles the 3.4 m tall 2D apple canopy structures to create an accurate map of the fruitlets on each tree. We show that this platform can measure the fruitlet load on an apple tree by scanning through both sides of the branch. The requirement of an overarching platform was justified since two-sided scans had a higher counting accuracy of 81.17% than one-sided scans at 73.7%. The system was also demonstrated to produce size estimates within 5.9% RMSE of their true size.

Social robots are becoming increasingly present in our daily lives and will continue to be integrated into society to help people with their daily routines. In this paper, we create a general motion dictionary for the NAO robot, to generate emotional gestures when the robot is interacting with humans. We implemented the motions in the context of a museum setting, wherein NAO interacts with visitors as a guide. We present a Motion Dictionary which integrates each gesture’s features and the corresponding emotions. By using the Choregraphe simulator to create the motions and validate them with a real robot, we intend to simplify and help with the generation of emotional gestures for human-robot interaction.

Specific emotion detection in written human language is a challenging problem in various research fields, including psychology, neuroscience, and computer science. Twitter is a suitable source for collecting large emotion datasets, as users have provided tweets with emotion hashtags (e.g., #fear, #anger, #sadness, #joy, #surprise, and #disgust) expressing their emotions. However, the criteria for data collection, i.e., the position of representative or synonymous emotion hashtags, remains unclear. Next to this unclarity, we assess the suitability of various machine learning (ML) algorithms for this purpose. In this study, we collected over five million tweets (n=5,645,139) with 24 emotion hashtags and investigated the efficacy of different criteria for collecting tweets. Contrary to previous research, we found that applying any position of representative emotion hashtags can achieve strong performance, rather than applying the last position of synonymous emotion hashtags. Our study shows that the RoBERTa-large transformer model outperforms deep learning algorithms and traditional ML algorithms in terms of specific emotion detection in tweets, especially when trained on a dataset with a balance between size and quality. We also found that larger datasets are more efficient for RoBERTa model training than smaller datasets. Along with these empirical contributions, we share the collected emotion dataset.

Abstract Tactics to determine the emotions of authors of texts such as Twitter messages often rely on multiple annotators who label relatively small data sets of text passages. An alternative method gathers large text databases that contain the authors’ self-reported emotions, to which artificial intelligence, machine learning, and natural language processing tools can be applied. Both approaches have strength and weaknesses. Emotions evaluated by a few human annotators are susceptible to idiosyncratic biases that reflect the characteristics of the annotators. But models based on large, self-reported emotion data sets may overlook subtle, social emotions that human annotators can recognize. In seeking to establish a means to train emotion detection models so that they can achieve good performance in different contexts, the current study proposes a novel transformer transfer learning approach that parallels human development stages: (1) detect emotions reported by the texts’ authors and (2) synchronize the model with social emotions identified in annotator-rated emotion data sets. The analysis, based on a large, novel, self-reported emotion data set (n = 3,654,544) and applied to 10 previously published data sets, shows that the transfer learning emotion model achieves relatively strong performance.

Clinical empathy has been associated with many positive outcomes, including patient trust and satisfaction. Physicians can demonstrate clinical empathy through verbal statements and non-verbal behaviors, such as head nodding. The use of verbal and non-verbal empathy behaviors by healthcare robots may also positively affect patient outcomes. The current study examined whether the use of robot verbal empathy statements and head nodding during a video recorded interaction between a healthcare robot and patient improved participant trust and satisfaction. One hundred participants took part in the experiment, online through Amazon Mechanical Turk. They were randoimnized to watch one of four videos depicting an interaction with a `patient' and a Nao robot that (1) either made empathetic or neutral statements, and (2) either nodded its head when listening to the patient or did not. Results showed that the use of empathetic statements by the healthcare robot significantly increased participant perceptions of robot empathy, trust and satisfaction, and reduced robot distrust. No significant findings were revealed in relation to robot head nodding. The positive effects of empathy statements support the model of Robot-Patient Communication, which theorizes that robot use of recommended clinical empathy behaviors can improve patient outcomes. The effects of healthcare robot nodding behavior needs to be further investigated.

Social robots are often critiqued as being too ‘robotic’ and unemotional. For affective human-robot interaction (HRI), robots must detect sentiment and express emotion and empathy in return. We explored the extent to which people can detect emotions, empathy and sentiment from speech expressed by a computer system, with a focus on changes in prosody (pitch, tone, volume) and how people identify sentiment from written text, compared to a sentiment analyzer. 89 participants identified empathy, emotion and sentiment from audio and text embedded in a survey. Empathy and sentiment were best expressed in the audio, while emotions were the most difficult detect (75%, 67% and 42% respectively). We found moderate agreement (70%) between the sentiment identified by the participants and the analyzer. There is potential for computer systems to express affect by using changes in prosody, as well as analyzing text to identify sentiment. This may help to further develop affective capabilities and appropriate responses in social robots, in order to avoid ‘robotic’ interactions. Future research should explore how to better express negative sentiment and emotions, while leveraging multi-modal approaches to HRI.

Participatory design includes stakeholders in the development of products intended to solve real-life challenges. Involving end users in the design of robots is vital for developing effective, useful, acceptable and user-friendly products that meet expectations, needs, and preferences. This four-year international project developed and evaluated a home-based robot for mood stabilization and cognitive improvement in older adults with mild cognitive impairment and age-related health needs. The daily-care robot was developed in collaboration with experts, carers, relatives, and older adults, through six phases. Two phases were dedicated to cognitive stimulation games. This paper provides a summary of the participatory design and mixed-methods evaluation processes undertaken to develop, refine, and test the robot. The final robot and games were acceptable to older adults, and useful for delivering stimulating activities and providing reminders for medication, health and wellbeing checks. Personalization is required to optimize human-robot interaction, and imagery and speech should be consistent with local users. Functions should be personalizable to accommodate individual health needs and preferences. This project highlights the importance of participatory design and testing robotics in end-user environments, as technical issues associated with long-term use were uncovered. Recommendations for future development and the design of assistive health robots are made.

COVID-19, particularly vaccines, have caused an ‘infodemic’ online; a rapid and vast spread of unreliable information. While vaccines can minimize the detrimental effects of COVID-19, misinformation, fearmongering, and ‘anti-vax’ movements have fostered opposition which is especially prevalent on Twitter. Understanding public emotions related to vaccines is an important, yet inconsistent, area of research. To resolve some of the inconsistencies in the field, we develop and apply two integrated emotion detection models to a longitudinal sample of COVID-19 vaccine related tweets (n = 823,748). Contrary to prior research, which concluded that positive emotions are the most dominant emotion (e.g., trust and happiness), the balanced emotion model (consisting of eight emotions) shows that fear (41 %) is the most dominant emotion. The extended emotion model (consisting of sixteen emotions) shows various negative emotions such as panic (27%), fear (22%), and shame (37%) as the dominant emotions in the tweet hashtag groups such as COVID-19, Vaccine, and Anti-vaxxers.

Cognitive stimulation games delivered on robots may be able to improve cognitive functioning and delay decline in older adults. However, little is known about older adults’ in-depth opinions of robot-delivered games, as current research primarily focuses on technical development and one-off use. This article explores the usability, acceptability, and perceptions of community-dwelling older adults towards cognitive games delivered on a robot that incorporated movable interactive blocks. Semi-structured interviews were conducted with participants at the end of a 12-week cognitive stimulation games intervention delivered entirely on robots. Participants were 10 older adults purposively sampled from two retirement villages. A framework analysis approach was used to code data to predefined themes related to technology acceptance (perceived benefits, satisfaction, and preference), and usability (effectiveness, efficiency, and satisfaction). Results indicated that cognitive games delivered on a robot may be a valuable addition to existing cognitive stimulation activities. The robot was considered easy to use and useful in improving cognitive functioning. Future developments should incorporate interactive gaming tools, the use of social anthropomorphic robots, contrasting colour schemes to accommodate macular degeneration, and cultural-specific imagery and language. This will help cater to the preferences and age-related health needs of older adults, to ultimately enhance usability and acceptability.

Task-oriented dialogue system has a vital role in Human-Robot Interaction (HRI). However, it has been developed based on conventional pipeline approach which has several drawbacks; expensive, time-consuming, and so on. Based on this approach, developers manually define a robot's behaviour such as gestures and facial expressions on the corresponding dialogue states. Recently, end-to-end learning of Recurrent Neural Networks (RNNs) is an attractive solution for the dialogue system. In this paper, we proposed a social robot system using end-to-end dialogue system in the context of hospital receptionist. We utilized Hybrid Code Network (HCN) as an end-to-end dialogue system and extended to select both response and gesture using RNN based gesture selector. We evaluate its performance with human users and compare the results with one of the conventional methods. Empirical result shows that the proposed method has benefits in terms of dialogue efficiency, which indicates how efficient users were in performing the given tasks with the help of the robot. Moreover, we achieved the same performance regarding the robot's gesture with the proposed method compared to manually defined gestures.

Service robots with social intelligence are starting to be integrated into our everyday lives. The robots are intended to help improve aspects of quality of life as well as improve efficiency. We are organizing an exciting workshop at HRI 2019 that is oriented towards sharing the ideas amongst participants with diverse backgrounds ranging from Human-Robot Interaction design, social intelligence, decision making, social psychology and aspects and robotic social skills. The purpose of this workshop is to explore how social robots can interact with humans socially and facilitate the integration of social robots into our daily lives. This workshop focuses on three social aspects of human-robot interaction: (1) technical implementation of social robots and products, (2) form, function and behavior, and (3) human behavior and expectations as a means to understand the social aspects of interacting with these robots and products.

A robot cannot be warm and friendly - or can it? To explore whether a robot can be a medical receptionist, we developed a robotic system for interacting with patients at a doctor's clinic, including acting friendly. We designed the robot to interact naturally with patients at the start and finish of a clinic visit. We investigated people's perceptions to the robot in a wizard-of-Oz study, where the participants interacted with the robot over four interactions. 40 participants evaluated the robot. The results indicate the participants thought the robot could be a friendly receptionist, especially after repeated interactions with the robot. However, the participants mainly thought the robot was friendly in a “professional” way, rather than a personal friend.

This paper presents a new version of robot receptionist system for healthcare facility environment. Our HealthBots consists of three subsystems: a receptionist robot system, a nurse assistant robot system, and a medical server. Our first version of receptionist robot, interacts with human at hospital reception, gives instructions to human verbally, but cannot understand what human says, so it uses a touch screen to get the response from human. In this paper, we design a receptionist robot that recognizes human face as well as speech, which enhances verbal interaction skill of robot. In addition, we design a reaction generation engine to generate appropriate reactive motions and speech. Moreover, we study which social skills are important to a hospital receptionist robot to enhance social interaction, such as friendliness and attention. We implemented perception modules, decision-making modules, and reaction modules to our HealthBots architecture, and did two case studies to find essential social skills for hospital receptionist robots.

The motivation of the work presented here is to create a component of a safety system based on 3D lidar sensors, specifically for industrial environments where some rules can be set for people who will be in close proximity to working robots. Specifically, the operating procedure that is put in place in the workplace is that all people must wear the provided high visibility clothing, which has retro-reflective strips attached. It is shown here that the retro-reflective strips provide a strong cue for pedestrian detection in the intensity data from a lidar sensor within a range of 4 metres. We present and compare multiple methods of exploiting this cue and provide a recommendation for how a safety system should be architected in order to best exploit the lidar intensity data in combination with more common approaches for detection of objects from the lidar range data. Amongst these detection methods is the use of neural networks, which present challenges for key components of standardized safety system development-in particular, for programming methodology control, interpretability of testing and diagnostic coverage. We propose methods for how to start to address these challenges and how to integrate neural networks into safety systems.

Mobile service robots have the potential to improve the efficiency of fruit production in orchards. One of the key tasks that such robots must perform is traversing the rows. Many of the past implementations of row following in orchards have been developed for rows where the trees appear like walls on both sides. Another orchard structure that is used is the pergola, where a sparse array of trunks and posts hold up a canopy, which resembles a ceiling. Navigation in pergola structured environments has received less attention. The variations in the pergola environment- including the presence of tall weeds, hanging branches, undulating terrain and varying geometry-make following the rows a challenging problem. This paper presents solutions for finding the row centreline in pergola structured environments, in the presence of real world variability. A 3D laser scanner is used to measure the positions of posts and trunks, amongst the other features in the pergola. From the extracted features, the mode gradient of nearest neighbours is used to find the row direction and hence the centreline. The practicality of the system is demonstrated by autonomously driving a mobile robot through over 5000 meters of a kiwifruit orchard with a pergola structure, using the row detection method. This method performs favourably compared to an existing method of row detection in kiwifruit orchards.

This paper presents a robot system for healthcare facility environments. Current healthcare robot systems do not address healthcare workflows well and our goal is to provide distributed, heterogeneous multiple robot systems that are capable of integrating with healthcare workflows and are easy to modify when workflow requirements change. The proposed system consists of three subsystems: a receptionist robot system, a nurse assistant robot system, and a medical server. The roles of the receptionist robot and the nurse assistant robot are to do tasks to help the human receptionist and nurse. The healthcare robots upload and download patient information through the medical server and provide data summaries to human care givers via a web interface. We developed these healthcare robot systems based on our new robotic software framework, which is designed to easily integrate different programming frameworks, and minimize the impact of framework differences and new versions. We test the functionalities of each healthcare robot system, evaluate the robot-robot collaboration, and present a case study.

Robots have been proposed to reduce the costs of the provision of healthcare in rural settings, but as yet little research has tested this. This study investigated the feasibility and cost-effectiveness of a robot measuring routine vital signs in a family medicine clinic in a rural setting. The length of patient consultations was compared before (N = 85 patients) and after a robot was deployed in the clinic (N = 48 patients). A Cafero touchscreen robot took the patient's vital signs prior to the consultation and transferred the results to the medical professional's computer. Time-savings were calculated in New Zealand dollar terms and compared to the costs of the robot and its maintenance. Results showed that consultation lengths were cut by 18% on average (3 minutes and 13 seconds). If 20% of the clinics' annual consultations were augmented with the robot this translates to a total annual savings of NZ$19075. The annual cost of the robot was calculated to be NZ$9400 overs 5 years. Present value calculations of Benefit Cost result in a Benefit Cost ratio of 2.3. These results support the cost-effectiveness of the robot in a rural medical clinic. Further research is needed to improve the services provided by the robot and test it in a larger trial.

This paper presents a possibility that we can teach what emotions a robot should express. For this, we design an artificial emotion decision system learned by feedbacks of users. The proposed system consists of three parts: a personality space with probability model, an emotion decision process, and an emotion learning process. (1) The personality space is designed based on the Five-Factor Model. In the personality space, we set up probability distributions of emotions. (2) The emotion decision process determines the probability values of emotions using the probability distributions of emotions in the personality space. (3) The emotion learning process updates the probability distributions by the feedbacks that are the teaching information from users; then, different probability values of emotions are determined. By applying to a humanoid robot system, we have verified the validity of the proposed system by being learned from two persons who have different personalities.

There are many theories about basic emotions, and we do not know which emotions are appropriate to use. Also, faces of robots are designed differently and require different ways to embody emotional expressions. Therefore, in this paper we address the appropriate emotions for facial expressions of EveR-4 H33, which is controlled by thirty-three motors for head system. EveR-4 H33 displays her facial expressions for certain emotions selected from typical basic emotion theories. Then, audiences at an exhibition evaluate her facial expressions, by enjoying a game of emotional correction. We analyze the results of the game, and decide appropriate emotions for EveR-4 H33.

This paper analyzes emotional expressions of an android head system according to gender and age. We use an EveR-4 H33 controlled by thirty-three motors for facial expression. EveR-4 H33 is a head system for an android face consists of three layers: a mechanical layer, an inner cover layer and an outer cover layer. Facial expressions of robots are different from the purposes of robots. In addition, feeling of emotional expressions is also different from humans depending on age, gender, etc. Therefore, we find the appropriate uses of EveR-4 H33 in this paper. EveR-4 H33 shows her facial expressions about some emotions. Then, audiences of exhibition evaluate her facial expressions by enjoying a game of emotional correction. We analyze the results of the game according to gender and age, and decide appropriate uses of EveR-4 H33.

This paper presents a behavior generating method for reflecting emotional probabilities. The proposed method consists of two processes: an emotion-behavior probability generating process and a unit behavior combination generating process. 1) In the emotion-behavior probability generating process, the emotional probabilities of behaviors are determined on the basis of user preferences in terms of the priorities of emotions. 2) In the unit behavior combination generating process, optimal behaviors are found by the simulated annealing algorithm. A final behavior is a set of selected parts of expressions. It is possible to not only reveal an abundance of expressions without one-to-one mapping relations between emotions and behaviors but also apply these expressions in the case of various robots. We have verified the diversity of emotional expression by applying the proposed method to two different robot systems, which are a cyber robot simulator and a real robot system.

This paper proposes an emotional behavior generator for emotional robots. The traditional methods for generating emotional behavior are not capable of expressing complex emotions, and lack the diversity of emotional expression and generality of system. To solve these problems, we propose a general emotional behavior generation module. It generates behavior combination expressing complex and phased emotions, using the concept of unit behavior and emotional marks. With behavior training sets of the module user, it generates the emotional matrix which represents expression abilities of unit behaviors. And with the emotional matrix, unit behaviors are combined into emotional expression behaviors using Simulated Annealing. To evaluate the results, we apply it to a robot simulator.

In this paper, we propose a new method for object tracking based on mean shift algorithm using a kernel which has the shape of the target object, and with probabilistic estimation of the orientation change and scale adaptation. The proposed method uses an object mask to construct a kernel which has the shape of the actual object for tracking. Orientation is adjusted using probabilistic estimation of orientation and scale is adapted using a newly proposed descriptor for scale. Tests results show that the proposed method is robust to background clutter and tracks objects very accurately.

This paper presents the design and implementation of a reconfigurable heterogeneous modular architecture for service robots. The proposed architecture has five key concepts which are different from conventional reconfigurable modular service robots; 1) easy and multiple assembly according to requirements of users, 2) hardware resource sharing system with other heterogeneous modules, 3) communication ability among all heterogeneous modules which have different operating systems, 4) automatic connection management system when a new module is attached, 5) automatic software upgrading system for new module software. We explain the three parts of the system architecture which meet the five concepts; mechanical architecture, software architecture, and connection architecture. To verify our architecture, we developed and evaluated a reconfigurable heterogeneous modular service robot by applying the proposed design.

This paper introduces an emotional behavior decision model for intelligent service robots. An emotional model should make different behavior decisions according to the purpose of the robots. We propose an emotional behavior decision model which can change the character of emotional model and make different behavior decisions although the situation and environment remain the same. We defined each emotional element such as reactive dynamics, internal dynamics, emotional dynamics, and behavior dynamics by state dynamic equations. The proposed system model is a linear system. If you want to add one external stimulus or behavior, you need to add just one dimensional vector to the matrix of external stimulus or behavior dynamics. The case of removing is same. The change of reactive dynamics, internal dynamics, emotional dynamics, and behavior dynamics also follows the same procedure. We implemented the proposed emotional behavior decision model and verified its performance.