Animal assisted intervention has been one of the effective natural therapeutic approaches, especially for individuals with autism. To increase the accessibility and reduce extra burden of care for the animals, Robotic-Animal Assisted Interventions (RAAI) has been proposed. However, the lack of natural behaviors is one of the key factors in limiting the feasibility with the current technology. This late-breaking report aims to build natural behavior models with data-driven approach, utilizing latest development of large language models (LLMs) to effectively analyze the data and build natural language-based models. Due to the reliance on LLM in this study, a key limitation is the lack of continuity in understanding. Frame images, as static representations, may not fully capture temporal dynamics. Future studies could address this limitation by integrating 3D-pose analysis, which would improve both continuity and contextual understanding.

Challenging behaviors in children with autism is a serious clinical condition, oftentimes leading to aggression or self-injurious actions. The Revised Family Observation Schedule 3 rd Edition (FOS-R-III) is an intensive and fine-grained scale used to observe and analyze the behaviors of individuals with autism, which facilitates the diagnosis and monitoring of autism severity. Previous AI-based approaches for automated behavior analysis in autism often focused on predicting facial expressions and body movements without generating a clinically meaningful scale, mostly utilizing visual information. In this study, we propose a deep-learning based algorithm with audio-visual multimodal-data clinically coded with the FOS-R-III, named AV-FOS model. Our proposed AV-FOS model leverages transformer-based structure and self-supervised learning to intelligently recognize Interaction Styles (IS) in the FOS-R-III scale from subjects' video recordings. This enables the automatic generation of the FOS-R-III measures with clinically acceptable accuracy. We explore the IS recognition using a multimodal large language model, GPT4V, with prompt engineering provided with FOS-R-III measure definitions as the baseline for this study and compare with other vision-based deep learning algorithms. We believe this research represents a significant advancement in autism research and clinical accessibility. The proposed AV-FOS and our FOS-R-III dataset will serve as a gateway toward the digital health era for future AI models related to autism.

Social robots have been demonstrated to support neurodivergent individuals in health and educational settings, but the roles and benefits of social robots in the everyday lives of neurodivergent people are underexplored. We investigated daily-life use cases of robot dogs for neurodivergent individuals through three co-design workshops over five weeks. The workshops included interactions between neurodivergent participants and robot dogs, followed by feedback sessions. Participants showed high acceptance levels towards robot dogs and envisioned use cases that fulfilled practical, emotional, and social needs. Some participants associated robotic failures with rejection, leading us to further explore the impacts and communication of failures. Results showed how robot dogs can provide opportunities for users to be in a caregiving role and engage in interpersonal interactions. We conclude by discussing how to leverage the potential benefits of social robots by designing for social opportunities and ways to design failures to mitigate potential harms for users.

Autistic individuals, or individuals diagnosed with autism spectrum disorder (ASD), often experience challenges in social relations and may experience increased stress or anxiety. Recent studies highlight the potential of socially assistive robots (SARs) as tools for robot-assisted interventions. However, current human-robot interaction (HRI) designs lack consideration for the diverse anxiety issues in individuals with ASD and the need for personalized, empathetic assistance. The question that persists is whether the most recent individualized and modular intervention program for autism can also be effectively implemented through the use of SARs. This study presents a novel HRI framework that incorporates psychological factors into a modular intervention aimed at reducing user’s anxiety levels. The developed system integrates techniques such as human pose estimation and motion imitation while also leveraging conversational agents. During the user study, users engaged in interactions with a humanoid robot equipped with multiple intervention modules, and their anxiety and perceptions of the robot’s empathy were assessed through questionnaires in a within-participants study design. Results show the implemented system effectively mitigated users’ anxiety by assessing the differences between pre-session and post-session scores. Additionally, our findings suggest that the observed change in empathetic scores may be a contributing factor to the reduction in anxiety levels. The study offers significant insights into the use of SARs for mental health support in autistic adolescents.

With the increase of prevalence in autism, the need for efficient public health support has been amplified. Socially-assistive robots (SARs) have been found effective in engaging and interacting with autistic children, however, the perception intelligence during interaction still needs more domain-specific knowledge in terms of understanding children’s behaviors. The Family Observation Schedule-Second Version (FOS-II) is one of the key methods in assessing parent-child interactions in developmental disabilities, yet its manual annotation demands considerable time and effort. This study proposes a multimodal artificial intelligence (AI) model using video and audio inputs for automated FOS-II annotation. Utilizing advanced deep learning for behavior recognition, this method offers rapid, cost-effective FOS-II scaling. It will thus enhance the capability of socially assistive robots to understand human behaviors and support the advancement of digital health research for children with autism. The visual perception in home settings are most likely based on uncontrolled environments, so it is crucial to develop algorithms that can robustly work with video-log data with uncontrolled quality. Ultimately, it aims to ease the burden on parents and caregivers, streamlining the monitoring and treatment of challenging behaviors in autism.

Using robots in therapy for children on the autism spectrum is a promising avenue for child-robot interaction, and one that has garnered significant interest from the research community. After preliminary interviews with stakeholders and evaluating music selections, twelve typically developing (TD) children and three children with Autism Spectrum Disorder (ASD) participated in an experiment where they played the dance freeze game to four songs in partnership with either a NAO robot or a human partner. Overall, there were significant differences between TD children and children with ASD (e.g., mimicry, dance quality, & game play). There were mixed results for TD children, but they tended to show greater engagement with the researcher. However, objective results for children with ASD showed greater attention and engagement while dancing with the robot. There was little difference in game performance between partners or songs for either group. However, upbeat music did encourage greater movement than calm music. Using a robot in a musical dance game for children with ASD appears to show the advantages and potential just as in previous research efforts. Implications and future research are discussed with the results.

The diagnosis of Autism Spectrum Disorder (ASD) in children is commonly accompanied by a diagnosis of sensory processing disorders. Abnormalities are usually reported in multiple sensory processing domains, showing a higher prevalence of unusual responses, particularly to tactile, auditory and visual stimuli. This paper discusses a novel robot-based framework designed to target sensory difficulties faced by children with ASD in a controlled setting. The setup consists of a number of sensory stations, together with two different robotic agents that navigate the stations and interact with the stimuli. These stimuli are designed to resemble real world scenarios that form a common part of one's everyday experiences. Given the strong interest of children with ASD in technology in general and robots in particular, we attempt to utilize our robotic platform to demonstrate socially acceptable responses to the stimuli in an interactive, pedagogical setting that encourages the child's social, motor and vocal skills, while providing a diverse sensory experience. A preliminary user study was conducted to evaluate the efficacy of the proposed framework, with a total of 18 participants (5 with ASD and 13 typically developing) between the ages of 4 and 12 years. We derive a measure of social engagement, based on which we evaluate the effectiveness of the robots and sensory stations in order to identify key design features that can improve social engagement in children.

In an eight-week STEAM education program for elementary school children, kids worked on musical theater projects with a variety of robots. The program included 4 modules about acting, dancing, music & sounds, and drawing. Twenty-five children grades K-5 participated in this program. Children were excited by the program and they demonstrated collaboration and peer-to-peer interactive learning. In the future, we plan to add more robust interaction and more science and engineering experiences to the program. This program is expected to promote STEM education in the informal learning environment by combining it with arts and design.

Robotic rescuers digging through rubble, fire-fighting drones flying over populated areas, robotic servers pouring hot coffee for you, and a nursing robot checking your vitals are all examples of current or near-future situations where humans and robots are expected to interact in a dangerous situation. Dangerous HRI is an as-yet understudied area of the field. We define dangerous HRI as situations where humans experience some amount of risk of bodily harm while interacting with robots. This interaction could take many forms, such as a bystander (e.g. when an autonomous car waits at a crossing for a pedestrian), as a recipient of robotic assistance (rescue robots), or as a teammate (like an autonomous robot working with a SWAT team). To facilitate better study of this area, the Dangerous HRI workshop brings together researchers who perform experiments with some risk of bodily harm to participants and discuss strategies for mitigating this risk while still maintaining validity of the experiment. This workshop does not aim to tackle the general problem of human safety around robots, but instead focused on guidelines for and experience from experimenters.

Dementia is a growing problem amongst elderly adults and the number of dementia patients is predicted to rise considerably in the coming years. While there is no cure for dementia, recent studies have suggested that exercise may have a positive effect on the cognitive function of dementia patients. We propose that a humanoid therapy robot is an effective tool for encouraging exercise in dementia patients. Such a robot will help address problems such as cost of care and shortage of healthcare workers. We have developed an interactive robotic system and conducted preliminary tests with a robot that encourages a user to engage in simple dance moves. The heart rate is used as feedback to decide which exercise move should be demonstrated. The results we have found are promising and we hope to continue this work via future studies.

The papers in this special section examine robotic technologies of the fourth industrial revolution or Industry 4.0 that will impact manufacturing industries. The concept of the fourth industrial revolution has drawn attention throughout the world and many efforts to define the concept in diverse fields have continued. Generally, the concept can be summarized as the technology convergence through hyper-intelligence and hyper-connectivity. The core technologies providing the thrust of the fourth industrial revolution, especially in the industrial informatics field, are Internet of Things, robotics, virtual reality, and artificial intelligence. As one of the most critical characteristics of the fourth industrial revolution technology is that the boundary between cyber space and physical space becomes unclear, innovations in industry and business initiate through the fusion of these two spaces. It is the robotic system that plays the key role as a physical medium linking cyber and physical spaces and even changing the physical space through direct interactions. In this sense, robotic system should be recognized as a crucial platform in performing tasks in the cyber-physical space.

Recently, image captioning has received much attention from the artificial-intelligent (AI) research community. Most of the current works follow the encoder-decoder machine translation model to automatically generate captions for images. However, most of these works used Convolutional Neural Network (CNN) as an image encoder and Recurrent Neural Network (RNN) as a decoder to generate the caption. In this paper, we propose a sequence-to-sequence model that uses RNN as an image encoder that follows the encoder-decoder machine translation model, such that the input to the model is a sequence of images that represents the objects in the image. These objects are ordered based on their order in the captions. We demonstrate the results of the model on Flickr30K dataset and compare the results with the state-ofthe-art methods that use the same dataset. The proposed model outperformed the state-of-the-art methods on all metrics.

As the technology needed to make robots robust and affordable draws ever nearer, human-robot interaction (HRI) research to make robots more useful and accessible to the general population becomes more crucial. In this study, 59 college students filled out an online survey soliciting their judgments regarding seven social robots based solely on appearance. Results suggest that participants prefer robots that resemble animals or humans over those that are intended to represent an imaginary creature or do not resemble a creature at all. Results are discussed based on social robot application and design features.

Physical embodiment of robots provides users with a social environment. To design social robots further to be accepted as our companions, we need to understand the essential factors and implement them and so, users get to bring them to their personal environments. To this aim, we focused on two important factors in robotic companionship: robot appearance (look) and emotional expression (feel). Twenty-one participants played an online game with the help from two humanoid robots, Nao (more human-like looking) and Darwin (less human-like looking). Participants interacted with each robot either with emotional words or without emotional words. Results show that only when the robot both looks more human-like and speaks with emotional expression, participants perceive it as their companion. Implications are discussed with future works.

Children with autism have a hard time both with expressing their emotions, and with understanding the emotions of others. As the population of children with autism increases, it is crucial we create effective therapeutic programs that will improve their communication skills. We present an interactive robotic system that delivers emotional and social behaviors for multi-sensory therapy for children with autism spectrum disorders. Our framework includes emotion-based robotic gestures and facial expressions, as well as vision and audio-based monitoring system for quantitative measurement of the interaction.

We present an interactive robotic framework that delivers emotional and social behaviors for multi-sensory therapy for children with autism spectrum disorders. Our framework includes emotion-based robotic gestures and facial expressions, as well as vision and audio-based monitoring system for quantitative measurement of the interaction. We also discuss the special aspects of interacting with children with autism with multi-sensory stimuli and the potentials of our approach for personalized therapies for social and behavioral learning.

We present a hap tic force-profile acquisition and display system for a surgical device that will aid medical students in developing and learning surgical skills for future procedures. The goal of this study is to extend the benefits of using a low-cost surgical simulator by capturing subtle hand movements and providing realistic force feedback, enabling more realistic interaction during virtual reality based simulation.

The goal of this research is to evaluate the benefits of using a low-cost Virtual Reality (VR) surgical training platform in bridging the gap between early surgical skills and effective performance in more advanced training and clinical settings. The specific aim of this study is to design and evaluate the efficacy of a low-cost virtual reality system as a precursor to improving surgical skills for novice trainees. A summary of our VR training system is presented, and training results from three pre-med students and five residents in a medical school are discussed that show preliminary evidence of the efficacy of the system.

Virtual reality (VR) surgical training can be a potentially useful method for improving practicing surgical skills. However, the current literature on VR training has not discussed the efficacy of VR systems that are useful outside of the training facility. As such, the goal of this study is to evaluate the benefits of using a low-cost VR simulation system for providing a method to increase the learning of surgical skills. Our pilot case focuses on laparoscopic cholecystectomy, which is one of the most common surgeries currently performed in the United States and is often used as the training case for laparoscopy due to its high frequency and perceived low risk. The specific aim of this study is to examine the efficacy of a low-cost haptic-based VR surgical simulator on improving practicing surgical skills, measured by the change in the learning effect of students.

This paper presents a robotic system that provides telepresence to the visually impaired by combining real-time haptic rendering with multi-modal interaction. A virtual-proxy based haptic rendering process using a RGB-D sensor is developed and integrated into a unified framework for control and feedback for the telepresence robot. We discuss the challenging problem of presenting environmental perception to a user with visual impairments and our solution for multi-modal interaction. We also explain the experimental design and protocols, and results with human subjects with and without visual impairments. Discussion on the performance of our system and our future goals are presented toward the end.

Robotic assistance through telepresence technology is an emerging area in aiding the visually impaired. By integrating the robotic perception of a remote environment and transferring it to a human user through haptic environmental feedback, the disabled user can increase one's capability to interact with remote environments through the telepresence robot. This paper presents a framework that integrates visual perception from heterogeneous vision sensors and enables real-time interactive haptic represent-ation of the real world through a mobile manipulation robotic system. Specifically, a set of multi-disciplinary algorithms such as stereo-vision processes, three-dimensional map building algorithms, and virtual-proxy haptic rendering processes are integrated into a unified framework to accomplish the goal of real-world haptic exploration successfully. Results of our framework in an indoor environment are displayed, and its performances are analyzed. Quantitative results are provided along with qualitative results through a set of human subject testing. Our future work includes real-time haptic fusion of multi-modal environmental perception and more extensive human subject testing in a prolonged experimental design.

In this paper, we discuss an approach for enabling students with a visual impairment (VI) to validate the program sequence of a robotic system operating in the real world. We introduce a method that enables the person with VI to feel their robot's movement as well as the environment in which the robot is traveling. The design includes a human-robot interaction framework that utilizes multi-modal feedback to transfer the environmental perception to a human user with VI. Haptic feedback and auditory feedback are selected as primary methods for user interaction. Using this multi-modal sensory feedback approach, participants are taught to program their own robot to accomplish varying navigation tasks. We discuss and analyze the implementation of the method as deployed during two summer camps for middle-school students with visual impairment.

In this paper, we discuss a coordinated haptic training architecture useful for transferring expertise in teleoperation-based manipulation between two human users. The objective is to construct a reality-based haptic interaction system for knowledge transfer by linking an expert's skill with robotic movement in real time. The benefits from this approach include 1) a representation of an expert's knowledge into a more compact and general form by learning from a minimized set of training samples, and 2) an increase in the capability of a novice user by coupling learned skills absorbed by a robotic system with haptic feedback. In order to evaluate our ideas and present the effectiveness of our paradigm, human handwriting is selected as our experiment of interest. For the learning algorithms, artificial neural network (ANN) and support vector machine (SVM) are utilized and their performances are compared. For the evaluation of the performance of the output of the learning modules, a modified Longest Common Subsequence (LCSS) algorithm is implemented. Results show that one or two experts' samples are sufficient for the generation of haptic training knowledge, which can successfully recreate manipulation motion with a robotic system and transfer haptic forces to an untrained user with a haptic device. Also in the case of handwriting comparison, the similarity measures result in up to an 88% match even with a minimized set of training samples.

Learning control strategies from examples has been identified as an important capability for many robotic systems. In this work we show how the learning process can be aided by autonomously filtering the training set provided to improve key properties of the learning process. Demonstrated with data gathered for manipulation tasks, the results herein show the improved performance when autonomous filtering is applied. The filtration method, with no prior knowledge of the task, was able to partition the training sets into sets almost equal to expertly labeled sets. In the case where the filter did not produce the same groupings as the expert user, the method still permitted a controller to be trained which demonstrated a success rate of 92%.

In this paper, we discuss a methodology that employs vision-based force guidance techniques for improving human performance with respect to a teleoperated manipulation system. The primary focus of the approach is to study the effectiveness of guidance forces in a haptic system to enable ease-of-use for human operators performing common manipulation activities necessary for achievement of everyday tasks. By designing force feedback signals constructed only from visual imagery data as input into a haptic device, we show the impact on human performance during the teleoperation sequence. The methodology is explained in detail, and results of implementation on object-centering and object-approaching tasks with our divided force guidance approach are presented.