Sign language is an essential communication tool, however, it can be highly challenging for non-deaf students to learn. We propose a Pepper robot based sign language teaching assistant called SignPepper, with the ability to communicate in both spoken and sign language. Using Whisper speech to text and Llama 3.3, SignPepper can engage in two way spoken lessons, with the ability to physically demonstrate signs to students. Furthermore, using 3D convolutional neural networks trained on sign language recognition, SignPepper can watch, analyze and give personalized feedback on students attempts at performing newly learned signs in real-time; including hand-based error localization.

This study introduces “Guide Dog AR,” an AR device inspired by the handle of a guide dog’s harness. The device and its associated content aim to provide visually impaired individuals with the experience of walking with a guide dog, serving as both a practical tool and a foundation for entertainment. The device offers an augmented walking experience through tactile and auditory sensations. The device was assessed with the factors "effectiveness" and "immersiveness." The evaluation for effectiveness involved creating a simulated virtual path. The level of immersion was evaluated based on how deeply users were engrossed in the augmented content provided by the device. The success rate was determined by how well users navigated this virtual path using the device’s feedback. In assessing immersion, in-depth interviews were conducted with visually impaired participants who experienced the virtual path. These interviews compared their experiences to actual walks with guide dogs, discussing similarities, differences, and overall immersion.

Multiple modalities can boost accuracy in the difficult task of Sign Language Recognition (SLR), however, each modality does not necessarily contribute the same quality of information. Current multi-modal approaches assign the same importance weightings to each modality, or set weightings based on unproven heuristics. This paper takes a systematic approach to find the optimal weights by performing grid search. Firstly, we create a multi-modal version of the RGB only WLASL100 data with additional hand crop and skeletal pose modalities. Secondly, we create a 3D CNN based weighted multi-modal sign language network (WMSLRnet). Finally, we run various grid searches to find the optimal weightings for each modality. We show that very minor adjustments in the weightings can have major effects on the final SLR accuracy. On WLASL100, we significantly outperform previous networks of similar design, and achieve high accuracy in SLR without highly complex pre-training schemes or extra data.

Aotearoa New Zealand has a strong and growing apple industry but struggles to access workers to complete skilled, seasonal tasks such as thinning. To ensure effective thinning and make informed decisions on a per-tree basis, it is crucial to accurately measure the crop load of individual apple trees. However, this task poses challenges due to the dense foliage that hides the fruitlets within the tree structure. In this paper, we introduce the vision system of an automated apple fruitlet thinning robot, developed to tackle the labor shortage issue. This paper presents the initial design, implementation, and evaluation specifics of the system. The platform straddles the 3.4 m tall 2D apple canopy structures to create an accurate map of the fruitlets on each tree. We show that this platform can measure the fruitlet load on an apple tree by scanning through both sides of the branch. The requirement of an overarching platform was justified since two-sided scans had a higher counting accuracy of 81.17% than one-sided scans at 73.7%. The system was also demonstrated to produce size estimates within 5.9% RMSE of their true size.

Social robots are becoming increasingly present in our daily lives and will continue to be integrated into society to help people with their daily routines. In this paper, we create a general motion dictionary for the NAO robot, to generate emotional gestures when the robot is interacting with humans. We implemented the motions in the context of a museum setting, wherein NAO interacts with visitors as a guide. We present a Motion Dictionary which integrates each gesture’s features and the corresponding emotions. By using the Choregraphe simulator to create the motions and validate them with a real robot, we intend to simplify and help with the generation of emotional gestures for human-robot interaction.

Specific emotion detection in written human language is a challenging problem in various research fields, including psychology, neuroscience, and computer science. Twitter is a suitable source for collecting large emotion datasets, as users have provided tweets with emotion hashtags (e.g., #fear, #anger, #sadness, #joy, #surprise, and #disgust) expressing their emotions. However, the criteria for data collection, i.e., the position of representative or synonymous emotion hashtags, remains unclear. Next to this unclarity, we assess the suitability of various machine learning (ML) algorithms for this purpose. In this study, we collected over five million tweets (n=5,645,139) with 24 emotion hashtags and investigated the efficacy of different criteria for collecting tweets. Contrary to previous research, we found that applying any position of representative emotion hashtags can achieve strong performance, rather than applying the last position of synonymous emotion hashtags. Our study shows that the RoBERTa-large transformer model outperforms deep learning algorithms and traditional ML algorithms in terms of specific emotion detection in tweets, especially when trained on a dataset with a balance between size and quality. We also found that larger datasets are more efficient for RoBERTa model training than smaller datasets. Along with these empirical contributions, we share the collected emotion dataset.

Abstract Tactics to determine the emotions of authors of texts such as Twitter messages often rely on multiple annotators who label relatively small data sets of text passages. An alternative method gathers large text databases that contain the authors’ self-reported emotions, to which artificial intelligence, machine learning, and natural language processing tools can be applied. Both approaches have strength and weaknesses. Emotions evaluated by a few human annotators are susceptible to idiosyncratic biases that reflect the characteristics of the annotators. But models based on large, self-reported emotion data sets may overlook subtle, social emotions that human annotators can recognize. In seeking to establish a means to train emotion detection models so that they can achieve good performance in different contexts, the current study proposes a novel transformer transfer learning approach that parallels human development stages: (1) detect emotions reported by the texts’ authors and (2) synchronize the model with social emotions identified in annotator-rated emotion data sets. The analysis, based on a large, novel, self-reported emotion data set (n = 3,654,544) and applied to 10 previously published data sets, shows that the transfer learning emotion model achieves relatively strong performance.

Social robots are often critiqued as being too ‘robotic’ and unemotional. For affective human-robot interaction (HRI), robots must detect sentiment and express emotion and empathy in return. We explored the extent to which people can detect emotions, empathy and sentiment from speech expressed by a computer system, with a focus on changes in prosody (pitch, tone, volume) and how people identify sentiment from written text, compared to a sentiment analyzer. 89 participants identified empathy, emotion and sentiment from audio and text embedded in a survey. Empathy and sentiment were best expressed in the audio, while emotions were the most difficult detect (75%, 67% and 42% respectively). We found moderate agreement (70%) between the sentiment identified by the participants and the analyzer. There is potential for computer systems to express affect by using changes in prosody, as well as analyzing text to identify sentiment. This may help to further develop affective capabilities and appropriate responses in social robots, in order to avoid ‘robotic’ interactions. Future research should explore how to better express negative sentiment and emotions, while leveraging multi-modal approaches to HRI.

Participatory design includes stakeholders in the development of products intended to solve real-life challenges. Involving end users in the design of robots is vital for developing effective, useful, acceptable and user-friendly products that meet expectations, needs, and preferences. This four-year international project developed and evaluated a home-based robot for mood stabilization and cognitive improvement in older adults with mild cognitive impairment and age-related health needs. The daily-care robot was developed in collaboration with experts, carers, relatives, and older adults, through six phases. Two phases were dedicated to cognitive stimulation games. This paper provides a summary of the participatory design and mixed-methods evaluation processes undertaken to develop, refine, and test the robot. The final robot and games were acceptable to older adults, and useful for delivering stimulating activities and providing reminders for medication, health and wellbeing checks. Personalization is required to optimize human-robot interaction, and imagery and speech should be consistent with local users. Functions should be personalizable to accommodate individual health needs and preferences. This project highlights the importance of participatory design and testing robotics in end-user environments, as technical issues associated with long-term use were uncovered. Recommendations for future development and the design of assistive health robots are made.

COVID-19, particularly vaccines, have caused an ‘infodemic’ online; a rapid and vast spread of unreliable information. While vaccines can minimize the detrimental effects of COVID-19, misinformation, fearmongering, and ‘anti-vax’ movements have fostered opposition which is especially prevalent on Twitter. Understanding public emotions related to vaccines is an important, yet inconsistent, area of research. To resolve some of the inconsistencies in the field, we develop and apply two integrated emotion detection models to a longitudinal sample of COVID-19 vaccine related tweets (n = 823,748). Contrary to prior research, which concluded that positive emotions are the most dominant emotion (e.g., trust and happiness), the balanced emotion model (consisting of eight emotions) shows that fear (41 %) is the most dominant emotion. The extended emotion model (consisting of sixteen emotions) shows various negative emotions such as panic (27%), fear (22%), and shame (37%) as the dominant emotions in the tweet hashtag groups such as COVID-19, Vaccine, and Anti-vaxxers.

Cognitive stimulation games delivered on robots may be able to improve cognitive functioning and delay decline in older adults. However, little is known about older adults’ in-depth opinions of robot-delivered games, as current research primarily focuses on technical development and one-off use. This article explores the usability, acceptability, and perceptions of community-dwelling older adults towards cognitive games delivered on a robot that incorporated movable interactive blocks. Semi-structured interviews were conducted with participants at the end of a 12-week cognitive stimulation games intervention delivered entirely on robots. Participants were 10 older adults purposively sampled from two retirement villages. A framework analysis approach was used to code data to predefined themes related to technology acceptance (perceived benefits, satisfaction, and preference), and usability (effectiveness, efficiency, and satisfaction). Results indicated that cognitive games delivered on a robot may be a valuable addition to existing cognitive stimulation activities. The robot was considered easy to use and useful in improving cognitive functioning. Future developments should incorporate interactive gaming tools, the use of social anthropomorphic robots, contrasting colour schemes to accommodate macular degeneration, and cultural-specific imagery and language. This will help cater to the preferences and age-related health needs of older adults, to ultimately enhance usability and acceptability.

A robot cannot be warm and friendly - or can it? To explore whether a robot can be a medical receptionist, we developed a robotic system for interacting with patients at a doctor's clinic, including acting friendly. We designed the robot to interact naturally with patients at the start and finish of a clinic visit. We investigated people's perceptions to the robot in a wizard-of-Oz study, where the participants interacted with the robot over four interactions. 40 participants evaluated the robot. The results indicate the participants thought the robot could be a friendly receptionist, especially after repeated interactions with the robot. However, the participants mainly thought the robot was friendly in a “professional” way, rather than a personal friend.

This paper presents a new version of robot receptionist system for healthcare facility environment. Our HealthBots consists of three subsystems: a receptionist robot system, a nurse assistant robot system, and a medical server. Our first version of receptionist robot, interacts with human at hospital reception, gives instructions to human verbally, but cannot understand what human says, so it uses a touch screen to get the response from human. In this paper, we design a receptionist robot that recognizes human face as well as speech, which enhances verbal interaction skill of robot. In addition, we design a reaction generation engine to generate appropriate reactive motions and speech. Moreover, we study which social skills are important to a hospital receptionist robot to enhance social interaction, such as friendliness and attention. We implemented perception modules, decision-making modules, and reaction modules to our HealthBots architecture, and did two case studies to find essential social skills for hospital receptionist robots.