As social robots become more integrated into daily life, understanding human perceptions of them is essential for effective design. This mixed factorial study examined how four emotions (happiness, sadness, anger, and relief) and a humanoid robot’s semantic-free utterances (SFUs) with male or female voices influence user perception. Forty-nine young participants listened to a fairy tale while the humanoid robot Pepper responded with gestures and emotional non-speech sounds. Emotion recognition accuracy and social perception were significantly higher in the Voice + Gesture condition than in the Gesture-Only condition. Happiness and sadness were most accurately recognized. Male voices elicited greater discomfort and perceptions of anthropomorphism, while male participants experienced more discomfort overall and female participants felt warmer toward Pepper. However, voice type and participant gender did not affect emotion recognition accuracy. The results provide insights for developing more effective and emotionally expressive social robots.

People have pointed to a connection between the creative arts and computing.In the present longitudinal pilot study we taught six programmers and six non-programmers how to read and write written crochet patterns with or without the accompanying crochet gestures.Half of programmers (three participants) and nonprogrammers (three participants) were taught with the gestures, while the other halves were not.Over two weeks we individually taught participants crochet during three separate 30 minute sessions.In a fourth session, we tested participants on crochet and elementary programming and algorithms.Test results showed that programmers and non-programmers performed better on average on both tests when they learned with gestures.We interviewed all participants afterwards; programmers provided examples of how crochet demonstrated elementary programming ideas, while nonprogrammers described what they thought about programming.Our empirical study provides evidence of embodied cognition and offers contributions towards developing novel teaching methods in computer science.

With the development of tools using generative artificial intelligence (GenAI) to create art, stakeholders cannot come to an agreement on the value of these works.In this study we uncovered the mixed opinions surrounding art made by AI.We developed two versions of a dance performance augmented by technology either with or without GenAI.For each version we informed audiences of how the performance was developed either before or after they had taken a survey on their perceptions of the performance.There were thirty-nine participants (13 males, 26 female) recruited and divided between the four performances.After the survey, we conducted focus groups with a subset of audience members.Results demonstrated that individuals were more inclined to attribute artistic merit to works made by GenAI when they were unaware its use.Our work contributes to the understanding of the design and reception of AI-made art.

Personal intelligent agents (IAs) are increasingly embedded in everyday life, a trend accelerated by generative AI technologies. Despite their growing presence, these agents often remain fragmented across different life domains and environments. This workshop explores how to design integrated IA ecosystems emphasizing continuity, coordination, and human-centered values. Participants with varied perspectives will collaboratively develop frameworks, scenarios, and guidelines for cohesive personal agent systems that enrich user experiences holistically. By examining factors that shape users' preferences for information integration or separation, we aim to inform the design of coherent, user-aligned multi-agent systems.

In-vehicle intelligent agents (IVIAs) have been developed to improve user experience in autonomous vehicles. Yet, the impact of the automation system on driver behavior and perception toward IVIAs is unclear. In this study, we conducted three experiments with 73 participants in a driving simulator to examine how automation system parameters (the level of automation system and IVIA features) influence driver attitudes, cognition, and behaviors when driving or riding in a simulated vehicle. We focused on subjective evaluations of driver-agent interaction and driver trust toward IVIAs to assess driver attitudes, driver situation awareness, and visual distraction to capture their cognition, and their driving performance to understand their behaviors. Our results show that the level of automation system affects drivers’ attitudes toward agent capabilities (e.g. perceived intelligence). Embodiment benefits are more pronounced with Level 5 systems, while speech style, in general, is more influential in determining affective aspects of user attitudes (e.g. Warmth, Likability). As the level of automation increases, drivers engage in more visual distractions. In addition, conversational speech style in general encouraged safer driving behaviors indicated by more stable lateral control under lower levels of automation. Our findings uncover the path of how system parameters affect driver behaviors through system evaluation and trust in agents. These findings have important implications for the development of cohesive user experiences in future transportation systems.

In-vehicle infotainment systems can cause various distractions, increasing the risk of car accidents. To address this problem, mid-air gesture systems have been introduced. This study investigated the potential of a novel interface that integrates a Head-Up Display (HUD) with auditory displays (spearcons: compressed speech) in a gesture-based menu navigation system to minimize visual distraction and improve driving and secondary task performance. The experiment involved 24 participants who navigated through 12 menu items using mid-air gestures while driving on a simulated road under four conditions: HUD (with, without spearcons) and Head-Down Display (HDD) (with, without spearcons). Results showed that the HUD condition significantly outperformed the HDD condition in participants’ level 1 situation awareness, perceived workload, menu navigation performance, and system usability. However, there were trade-offs on visual fixation duration on the menu, and lane deviation. These findings will guide future research in developing safer and more effective HUD-supported in-vehicle gesture interaction systems.

Motion sickness in passengers of automated vehicles could prevent users from taking advantage of the automation features, as well as lead to unsafe takeover conditions. To conduct motion sickness research, the first step requires motion sickness induction, and its verification. In the current study, twenty-nine participants experienced being a passenger in a fully self-driving vehicle on a motion-based driving simulator. A Control route designed to mimic typical driving conditions on a driving simulator, and a Treatment route that produced large lateral accelerations were tested. The Treatment route was also tested with a non-driving related task (NDRT). Results showed that exposure duration and increased lateral accelerations significantly increased motion sickness. Addition of the cognitively demanding NDRT led to a reduction in motion sickness, suggesting the possibility of a masking effect when engaged in an NDRT in an automated vehicle. Overall, the designed Treatment route was verified to successfully induce motion sickness.

Workers in outdoor settings encounter numerous constraints, such as weather conditions, safety-related issues, Wi-Fi connectivity, and frequent mobility. These constraints significantly influence their work efficiency, job satisfaction, safety, and well-being. Despite the development of advanced artificial intelligence, there is still much room for improvement in utilizing such technology for workers in outdoor environments. The present study addresses this gap by conducting a systematic review to understand the state of the art regarding the functions, agent types, technologies, evaluation methods, and dependent variables associated with intelligent agents for outdoor workers. The goal of the present study is to provide research directions and guidelines for the development of intelligent agents by addressing research questions, which aim to meet the specific needs of outdoor workers, ultimately contributing to improved working conditions and enhanced safety.

Over the past several decades, robots have gradually infiltrated our daily lives, evolving from tools to social companions. With more robots and artificial intelligence agents becoming capable of engaging in social and emotional interactions, investigating the dynamics behind these relationships is becoming increasingly relevant. Despite increasing interests, little research has conducted systematic research to unpack their underlying mechanisms. Utilizing the PRISMA framework, this review systematically examined attachment within Human-Robot Interaction (HRI) from 30 studies, by synthesizing research on human-robot emotional bonds from the last 15 years. This review highlights the need of integrating traditional psychological attachment theories to deepen our comprehension of these relationships. It also identifies gaps in current methodologies and suggests directions for future research, emphasizing the importance of grounding these studies in established psychological frameworks to better understand and facilitate these evolving human-robot relationships.

With the advancements of machine learning and AI technologies, robots have been more widely used in our everyday life and they have also been used in education. The present study introduces a 12-week child-robot theater afterschool program designed to promote science, technology, engineering, and mathematics (STEM) education with art elements (STEAM) for elementary students using social robots. Four modules were designed to introduce robot mechanisms as well as arts: Acting (anthropomorphism), Dance (robot movements), Music and Sounds (music composition), and Drawing (robot art). These modules provided children with basic knowledge about robotics and STEM and guided children to create a live robot theater play. A total of 16 students participated in the program, and 11 of them were involved in completing questionnaires and interviews regarding their perceptions towards robots, STEAM, and the afterschool program. Four afterschool program teachers participated in interviews, reflecting their perceptions of the program and observations of children’s experiences during the program. Our findings suggest that the present program effectively maintained children’s engagement and improved their interest in STEAM by connecting social robots and theater production. We conclude with design guidelines and recommendations for future research and programs.

Recent technological advancements have enabled the development of smarter (more automated) and more intelligent (adaptable) environments. To understand what factors lead users to reject or adopt Intelligent Environments (IEs), we reviewed nine prominent technology adoption theories. We conducted a literature review to investigate the acceptance and adoption of different types of IEs. We found that perceived usefulness, ease of use, perceived control or self-efficacy, affect and enjoyment, and perceived risks are the common factors across the studies explaining the adoption of IEs. However, shortcomings in the design and methods of the reviewed studies present major concerns in the generalizability and application of existing theories to emerging IEs. We identify eight lacunae in the existing literature and propose a new conceptual model for explaining the adoption of IEs. Through this study, we contribute to the formulation of the theoretical background for the successful introduction of IEs and their integration into users’ everyday life.

With the embodied interaction paradigm, research on human emotions has rapidly increased. In parallel, the advent of artificial intelligence and automated technologies has spurred research on trust towards interactive systems. However, little research has directly investigated the effects of emotions on trust in the context of technology use. The present paper surveyed empirical studies using the PRISMA framework. After briefly introducing emotional effects on cognitive processes, twenty-nine studies were systematically analyzed. In many papers positive emotions or empathically congruent systems led to higher trust. Some studies indicated that emotions can be a mediator between different factors and trust, whereas other studies showed only partial effects depending on different users or situations. Note that some research showed null effects or even negative effects (backfire effects) because emotional systems can be perceived as sarcastic or uncanny. In addition to the pervasive mood congruent effect or emotional contagion, various psychological mechanisms and theories were identified, such as entitativity, cognitive appraisal, or affect infusion model. Considerations for future design and research are discussed with results. This survey paper is expected to deepen the theoretical aspects of emotional effects on trust towards diverse technologies (robots, agents, or other interactive systems) and provide practical design directions.

Research shows that anger has a negative impact on cognition due to the rumination effect and in the context of driving, anger negatively impacts situation awareness, driving performance, and road safety. In-vehicle agents are capable of mitigating the effects of anger and subsequent effects on driving behavior. Language is another important aspect that influences information processing and human behavior during social interactions. This study aimed to explore the effects of the language of in-vehicle agents on angry drivers’ situation awareness, driving performance, and subjective perception by conducting a within-subject driving simulator study. Twenty four young drivers drove three different laps in a level 3 automated vehicle with a native-language speaking agent (Hindi or Chinese), second-language speaking agent (English) and no agent. The results of this study are indicative of the importance of native language processing in the context of driving. The use of the participants’ native language resulted in improved driving performance and heightened situation awareness. The participants preferred the native language agent over the other conditions and also expressed the need to control the state of the in-vehicle agent. The study results and discussions have theoretical and practical design implications and are expected to help foster future work in this domain.

The introduction of highly automated driving systems is expected to significantly change in-vehicle interactions, creating opportunities for the design of novel use cases and interactions for occupants. In this study, we sought to identify and extract these novel use cases and determine preliminary auditory display recommendations for these novel situations. We developed and generated use cases for level 4 automated vehicles through an expert workshop (N = 17) and online focus group interviews (N = 12). Most of the use cases we generated were then tested, apart from meditation, and user opinions were collected in a driving simulator study (N = 20). Results indicated participants were interested in functions that support their experience with both driving and non-driving related interactions in highly automated vehicles. Three categories of use cases for level 4 automated vehicles were developed: driving automation use cases, immersion use cases, and in-vehicle notification use cases. For the driving simulator study, we tested three display modalities for interaction with drivers: visual alert only, non-speech with visual, and speech with visual. In terms of situation awareness (SA), the non-speech with visual display was associated with significantly better SA for the use case consisting of a planned increase in automation level than the speech-with visual display. This study will provide guidance on sonification design to advance user experiences in highly automated vehicles.

While the needs and care for children on the autism spectrum have been widely investigated, the intervention and services available to autistic adults have been overlooked for a long time. This survey paper reviewed 32 articles that described and evaluated assistive technologies that have been developed and evaluated through a complete circle of interactive product design from ideation, prototype, and user evaluation. These assistive technologies aim to improve independence and living quality in autistic adults. We extracted information from the perspective of requirement gathering, technology designing, and effectiveness of evaluation in the design cycle. We found a general lack of requirements-driven design, and the evaluation process was not standardized either. The lack of requirement gathering results in designs purely based on existing literature without targeting actual user needs. Our synthesis of included paper contributes to developing iterative design considerations in assistive technologies for autistic adults. We also suggest that assistive technologies for autistic adults shift some attention from assisting only autistic adults who require at least substantial support to embracing also those who have been living independently but rather have difficulties in social interaction. Assistive technologies for them have the potentials to help them consolidate and enhance their experiences in independent living.

Research has identified applications of handheld-based VR, which utilizes handheld displays or mobile devices, for developing systems that involve users in mixed reality (MR) without the need for head-worn displays (HWDs). Such systems can potentially accommodate large groups of users participating in MR. However, we lack an understanding of how group sizes and interaction methods affect the user experience. In this paper, we aim to advance our understanding of handheld-based MR in the context of multiplayer, co-located games. We conducted a study (N = 38) to understand how user experiences vary by group size (2, 4, and 8) and interaction method (proximity-based or pointing-based). For our experiment, we implemented a multiuser experience for up to ten users. We found that proximity-based interaction that encouraged dynamic movement positively affected social presence and physical/temporal workload. In bigger group settings, participants felt less challenged and less positive. Individuals had varying preferences for group size and interaction type. The findings of the study will advance our understanding of the design space for handheld-based MR in terms of group sizes and interaction schemes. To make our contributions explicit, we conclude our paper with design implications that can inform user experience design in handheld-based mixed reality contexts.

Handheld devices have become an inclusive alternative to head-mounted displays in virtual reality (VR) environments, enhancing accessibility and allowing cross-device collaboration. Object manipulation techniques in 3D space with handheld devices, such as those in handheld augmented reality (AR), have been typically evaluated in table-top scale and we currently lack an understanding of how these techniques perform in larger scale environments. We conducted two studies, each with 30 participants, to investigate how different techniques impact usability and performance for room-scale handheld VR object translations. We compared three translation techniques that are similar to commonly studied techniques in handheld AR: 3DSlide, VirtualGrasp, and Joystick. We also examined the effects of target size, target distance, and user mobility conditions (stationary vs. moving). Results indicated that the Joystick technique, which allowed translation in relation to the user's perspective, was the fastest and most preferred, without difference in precision. Our findings provide insights for designing room-scale handheld VR systems, with potential implications for mixed reality systems involving handheld devices.

Using in-vehicle infotainment systems degrades driving performance and increases crash risk. To address this, we developed air gesture interfaces using various auditory displays. Thirty-two participants drove a simulator with air-gesture menu navigation tasks. A 4x2 mixed-model design was used to explore the effects of auditory displays as a within-subjects variable (earcons, auditory icons, spearcons, and no-sound) and menu-generation interfaces as a between-subjects variable (fixed and adaptive) on driving performance, secondary task performance, eye glance, and user experience. The adaptive condition centered the menu around the user’s hand position at the moment of activation, whereas the fixed condition located the menu always at the same position. Results demonstrated that spearcons provided the least visual distraction, least workload, best system usability and was favored by participants; and that fixed menu generation outperformed adaptive menu generation in driving safety and secondary task performance. Findings will inform design guidelines for in-vehicle air-gesture interaction systems.

Cro-Create is a crochet gesture recognition sonifier for individual and collaborative use. In single user mode, Cro-Create directly scales and maps numerical palm orientation values detected by a motion sensor to sound. In dual user mode, the system affords users with additional auditory feedback by detecting when two users’ gestures are synchronized by segmenting the gestural procedure of making a stitch in crochet into three stages and utilizing a dynamic time warping algorithm to classify and recognize these stages; when the system determines that both users have produced the same gesture, the sonification is complemented by a distinguishable chord. Through this demonstration we introduce our tool to share procedural state for the physical craftmaking process, crochet, through sound.

Accepted version

Recent hand tracking systems and wearable haptic devices have contributed to enhancing user experience in virtual environments. However, controllers are still prevalent in virtual reality (VR) games as a main interaction device. Also, haptic devices are rare and not widely accepted by users. In this article, we examined the effects of interaction methods and vibrotactile feedback on users’ sense of presence, engagement, usability, and task performance in a VR game. In our experiment, 36 participants wearing VR goggle played a rhythm game under three conditions: (1) VR controllers, (2) hand tracking without vibrotactile feedback, and (3) hand tracking with vibrotactile feedback at fingertips through the gloves we developed. Results showed that hand tracking improved presence, engagement, usability, and performance, compared to the VR controller. Furthermore, vibrotactile feedback enhanced presence and engagement even more clearly. Results are discussed with the components of VR user experience and practical design guidelines.

Advances in the fields of data processing and sonification have been applied to transcribe a variety of visual experiences into an auditory format. Although image sonification examples exist, the application of these principles to visual art has not been examined thoroughly. We sought to develop and evaluate a set of guidelines for the sonification of visual artworks. Through conducting expert interviews (N = 11), we created an initial sonification algorithm that accounts for art style, lightness, and color diversity to modulate the sonified output in terms of tempo and pitch. This algorithm was evaluated through user evaluations (N = 22). User study responses supported expert interview findings, the notion that sonification can be designed to match the experience of viewing an artwork, and showed interesting interaction effects among art styles, visual components, and musical parameters. We suggest the proposed guidelines can augment visitor experiences at art exhibits and provide the basis for further experimentation.

As robots have become more pervasive in our everyday life, social aspects of robots have attracted researchers’ attention. Because emotions play a crucial role in social interactions, research has been conducted on conveying emotions via speech. Our study sought to investigate the synchronization of multimodal interaction in human-robot interaction (HRI). We conducted a within-subjects exploratory study with 40 participants to investigate the effects of non-speech sounds (natural voice, synthesized voice, musical sound, and no sound) and basic emotions (anger, fear, happiness, sadness, and surprise) on user perception with emotional body gestures of an anthropomorphic robot (Pepper). While listening to a fairytale with the participant, a humanoid robot responded to the story with recorded emotional non-speech sounds and gestures. Participants showed significantly higher emotion recognition accuracy from the natural voice than from other sounds. The confusion matrix showed that happiness and sadness had the highest emotion recognition accuracy, which is in line with previous research. The natural voice also induced higher trust, naturalness, and preference compared to other sounds. Interestingly, the musical sound mostly showed lower perception ratings, even compared to no sound. Results are discussed with design guidelines for emotional cues from social robots and future research directions.

The use of social robots has recently been investigated in various areas, including STEM (Science, Technology, Engineering, and Mathematics) education and artistic performances. To inform children of the seriousness of climate change and awareness that they can make change, we created the Robot Musical Theater performance. In this project, natural elements (wind, earth, fire, and water) were anthropomorphized and represented by humanoid robots (Pepper, Milo, and Nao). The robots were designed to motivate audience to participate in the action to prevent climate change. Because of COVID, only fourteen visitors as a single group were allowed to participate in real-time and posted to YouTube, where at the time of submission, 141 people have viewed the performance. The participants provided positive comments on the performance and showed their willingness to participate in the movement to prevent climate change, and expressed their further interest in STEM learning. This performance is expected to contribute to enhancing informal STEM and robotics learning, as well as advancing robotic arts.

In semi-automated vehicles, non-speech sounds have been prevalently used as auditory displays for control transitions since these sounds convey urgency well. However, there are no standards of specifications for warning sounds so that diverse non-speech sounds are being employed. To shed light on this, the effects of different non-speech auditory warnings on driver performance were investigated and quantified through the experimental study and human performance modeling approaches. Twenty-four young drivers drove in the driving simulator and experienced both handover and takeover transitions between manual and automated modes while performing a secondary task. The reaction times for handover and takeover, mental workload, and subjective responses were reported. Overall, a traditional warning sound with many repetitions and an indicator sound with decreasing polarity outperformed and were preferred. Additionally, a mathematical model, using the Queuing Network-Model Human Processor (QN-MHP) framework, was applied to quantify the effects of auditory warnings’ acoustic characteristics on drivers’ reaction times in response to takeover request displays. The acoustic characteristics, including the fundamental frequency, the number of repetitions, and the range of dominant frequencies were utilized in modeling. The model was able to explain 99.7% of the experimental data with a root mean square error (RMSE) of 0.148. The present study can contribute to establishing standards and design guidelines for takeover request displays in semi-automated vehicles.

The introduction of fully automated vehicles (FAVs) will change user experiences (UX) in personal transportation. In order for FAVs to become a life enhancing technology, it is required to design vehicular applications and user interfaces based on users’ expectations. To this end, we investigated user needs and design requirements. First, we elicited design taxonomy and use cases through literature review and trend analysis. Using these materials, expert interviews (N = 9) and focus group interviews (N = 10) were conducted. Through the qualitative analysis, we obtained twelve categories of user needs and devised design requirements based on the updated design taxonomy. While some of them have been an extension of current experiences in manual driving, completely new demands have also emerged within FAVs. Our findings contribute to designing UX in FAVs by satisfying users’ expectations and key values that can guide designers.

Automated vehicles (AVs) are the next wave of evolution in the transportation industry, but the progress towards increased levels of automation faces several challenges. One of the major problems, that gets overlooked, is motion sickness. As more drivers become passengers engaging in ‘passenger tasks’, it will lead to greater occurrences of motion sickness, preventing AVs from providing their true benefit to society. In an attempt to encourage more researchers to solve the problem of motion sickness in AVs, this study conducted a literature review following the PRISMA framework to identify the latest research trends and methodologies. Based on the findings and limitations in the existing literature, this study suggests a bird's-eye-view research framework consisting of causation, induction, measurement, and mitigation techniques, that researchers and early practitioners can utilize to conduct research in this field. Furthermore, the paper highlights future research directions in mitigation techniques to combat motion sickness in AVs.

Research shows that emotions have a substantial influence on the cognitive processes of humans and in the context of driving, can negatively influence driving performance. Drivers’ interaction with in-vehicle agents can improve their emotional state and can lead to increased road safety. Language is another important aspect that influences human behavior and information processing. This study aims to explore the effects of native and secondary-language processing on emotional drivers’ situation awareness, driving performance, and subjective perception by conducting a within-subject simulation study. Twenty-four young drivers drove three different laps with a native-language speaking agent, secondary-language speaking agent, and no agent. The study results are indicative of the importance of native-language processing in the context of driving. Native-language agent condition resulted in improved driving performance and heightened situation awareness. The study results and discussions have theoretical and practical design implications and are expected to help foster future work in this domain.

Speech style and embodiment are two widely researched characteristics of in-vehicle intelligent agents (IVIAs). This study aimed to investigate the influence of speech style (informative vs. conversational) and embodiment (voice-only vs. robot) and their interaction effects on driver-agent interaction. We conducted a driving simulator experiment, where 24 young drivers experienced four different fully autonomous driving scenarios, accompanied by four types of agents each, and completed subjective questionnaires about their perception towards the agents. Results showed that both conversational agents and robot agents promoted drivers' likability and perceived warmth. These two features also demonstrated independent impacts. Conversational agents received higher anthropomorphism and animacy scores, while robot agents received higher competence and lower perceived workload scores. The pupillometry indicated that drivers were more engaged while accompanied by conversational agents. Our findings are able to provide insights on applying different features to IVIAs to fulfill various user needs in highly intelligent autonomous vehicles.

Using robots in therapy for children on the autism spectrum is a promising avenue for child-robot interaction, and one that has garnered significant interest from the research community. After preliminary interviews with stakeholders and evaluating music selections, twelve typically developing (TD) children and three children with Autism Spectrum Disorder (ASD) participated in an experiment where they played the dance freeze game to four songs in partnership with either a NAO robot or a human partner. Overall, there were significant differences between TD children and children with ASD (e.g., mimicry, dance quality, & game play). There were mixed results for TD children, but they tended to show greater engagement with the researcher. However, objective results for children with ASD showed greater attention and engagement while dancing with the robot. There was little difference in game performance between partners or songs for either group. However, upbeat music did encourage greater movement than calm music. Using a robot in a musical dance game for children with ASD appears to show the advantages and potential just as in previous research efforts. Implications and future research are discussed with the results.

As semi-automated vehicles get to have the ability to drive themselves, it is important (1) to explore drivers’ affective states which may influence takeover performance and (2) to design optimized control transition displays to warn drivers to take control back from the vehicles. The present study investigated the influence of anger on drivers’ takeover reaction time and quality, with varying urgency of auditory takeover request displays. Using a driving simulator, 36 participants experienced takeover scenarios in a semi-automated vehicle with a secondary task (game). Higher frequency and more repetitions of the auditory displays led to faster takeover reaction times, but there was no difference between angry and neutral drivers. For takeover quality, angry drivers drove faster, took longer to change lanes and had lower steering wheel angles than neutral drivers, which made riskier driving. Results are discussed with the necessity of affect research and display design guidelines in automated vehicles.

There has been much research on facial affect detection, but many of them fall short on accurately identifying expressions, due to changes in illumination, occlusion, or noise in uncontrolled environments. Also, not much research has been conducted on implementing the algorithms using multiple datasets, varying the size of dataset and the dimension of each image in the dataset. Our ultimate goal is to develop an optimized algorithm that can be used for real-time affect detection in automated vehicles. To this end, in this study we implemented the facial affect detection algorithms with various datasets and conducted a comparative analysis of performance across the algorithms. The algorithms implemented in the study included a Convolutional Neural Network (CNN) in Tensorflow, FaceNet using Transfer Learning, and Capsule Network. Each of these algorithms was trained using the three datasets (FER2013, CK+, and Ohio) to get the predicted results. The Capsule Network showed the best detection accuracy (99.3%) with the CK+ dataset. Results are discussed with implications and future work.

The diagnosis of Autism Spectrum Disorder (ASD) in children is commonly accompanied by a diagnosis of sensory processing disorders. Abnormalities are usually reported in multiple sensory processing domains, showing a higher prevalence of unusual responses, particularly to tactile, auditory and visual stimuli. This paper discusses a novel robot-based framework designed to target sensory difficulties faced by children with ASD in a controlled setting. The setup consists of a number of sensory stations, together with two different robotic agents that navigate the stations and interact with the stimuli. These stimuli are designed to resemble real world scenarios that form a common part of one's everyday experiences. Given the strong interest of children with ASD in technology in general and robots in particular, we attempt to utilize our robotic platform to demonstrate socially acceptable responses to the stimuli in an interactive, pedagogical setting that encourages the child's social, motor and vocal skills, while providing a diverse sensory experience. A preliminary user study was conducted to evaluate the efficacy of the proposed framework, with a total of 18 participants (5 with ASD and 13 typically developing) between the ages of 4 and 12 years. We derive a measure of social engagement, based on which we evaluate the effectiveness of the robots and sensory stations in order to identify key design features that can improve social engagement in children.

In an eight-week STEAM education program for elementary school children, kids worked on musical theater projects with a variety of robots. The program included 4 modules about acting, dancing, music & sounds, and drawing. Twenty-five children grades K-5 participated in this program. Children were excited by the program and they demonstrated collaboration and peer-to-peer interactive learning. In the future, we plan to add more robust interaction and more science and engineering experiences to the program. This program is expected to promote STEM education in the informal learning environment by combining it with arts and design.

Robotic rescuers digging through rubble, fire-fighting drones flying over populated areas, robotic servers pouring hot coffee for you, and a nursing robot checking your vitals are all examples of current or near-future situations where humans and robots are expected to interact in a dangerous situation. Dangerous HRI is an as-yet understudied area of the field. We define dangerous HRI as situations where humans experience some amount of risk of bodily harm while interacting with robots. This interaction could take many forms, such as a bystander (e.g. when an autonomous car waits at a crossing for a pedestrian), as a recipient of robotic assistance (rescue robots), or as a teammate (like an autonomous robot working with a SWAT team). To facilitate better study of this area, the Dangerous HRI workshop brings together researchers who perform experiments with some risk of bodily harm to participants and discuss strategies for mitigating this risk while still maintaining validity of the experiment. This workshop does not aim to tackle the general problem of human safety around robots, but instead focused on guidelines for and experience from experimenters.

As the technology needed to make robots robust and affordable draws ever nearer, human-robot interaction (HRI) research to make robots more useful and accessible to the general population becomes more crucial. In this study, 59 college students filled out an online survey soliciting their judgments regarding seven social robots based solely on appearance. Results suggest that participants prefer robots that resemble animals or humans over those that are intended to represent an imaginary creature or do not resemble a creature at all. Results are discussed based on social robot application and design features.

Physical embodiment of robots provides users with a social environment. To design social robots further to be accepted as our companions, we need to understand the essential factors and implement them and so, users get to bring them to their personal environments. To this aim, we focused on two important factors in robotic companionship: robot appearance (look) and emotional expression (feel). Twenty-one participants played an online game with the help from two humanoid robots, Nao (more human-like looking) and Darwin (less human-like looking). Participants interacted with each robot either with emotional words or without emotional words. Results show that only when the robot both looks more human-like and speaks with emotional expression, participants perceive it as their companion. Implications are discussed with future works.

Children with autism have a hard time both with expressing their emotions, and with understanding the emotions of others. As the population of children with autism increases, it is crucial we create effective therapeutic programs that will improve their communication skills. We present an interactive robotic system that delivers emotional and social behaviors for multi-sensory therapy for children with autism spectrum disorders. Our framework includes emotion-based robotic gestures and facial expressions, as well as vision and audio-based monitoring system for quantitative measurement of the interaction.

We present an interactive robotic framework that delivers emotional and social behaviors for multi-sensory therapy for children with autism spectrum disorders. Our framework includes emotion-based robotic gestures and facial expressions, as well as vision and audio-based monitoring system for quantitative measurement of the interaction. We also discuss the special aspects of interacting with children with autism with multi-sensory stimuli and the potentials of our approach for personalized therapies for social and behavioral learning.

We have tried to promote STEM (Science, Technology, Engineering, and Math) education for underserved students using interactive robots. As an advanced attempt to integrate art and design into STEM education (i.e., STEAM), in the present paper we introduce our afterschool program in which elementary students create live theatre using multiple robots as actors. We hope to receive feedback and comments on our afterschool curriculum and case study, and thus, we can run better sessions at schools and make a standardized protocol regarding this robot actors approach.

While driving research on affect has mostly focused on anger and road rage, there has been little empirical research on other affective states. Affect researchers widely acknowledge the “sadder but wiser” phenomenon, but there is little evidence if this tendency can be applied to the driving environment as well. The objective of the present study is to empirically test whether sadness enhances driving performance as the sadder but wiser notion might predict or sadness impairs driving performance as its negative valence or low arousal dimension might predict. The study consists of a simulated driving experiment with induced anger, sadness, and neutral affect to examine how anger and sadness influence driving-related risk perception, driving performance, and perceived workload. Sixty-one young drivers drove under three different road conditions with either induced anger, sadness, or neutral affect conditions. After affect induction, there was no difference in subjective risk perception across three affect conditions. However, participants in both affect conditions showed significantly more errors and took longer driving time than those in the neutral condition. Only participants with induced anger reported significantly higher physical workload and frustration than participants with neutral affect. Results are discussed in terms of affect mechanisms, design directions for the in-vehicle affect mitigation system, and limitations of the study.

Auditory display research for driving has mainly examined a limited range of tasks (e.g., collision warnings, cell phone tasks). In contrast, the goal of this project was to evaluate the effectiveness of enhanced auditory menu cues in a simulated driving context. The advanced auditory cues of “spearcons” (compressed speech cues) and “spindex” (a speech-based index cue) were predicted to improve both menu navigation and driving. Two experiments used a dual task paradigm in which users selected songs on the vehicle’s infotainment system. In Experiment 1, 24 undergraduates played a simple, perceptual-motor ball-catching game (the primary task; a surrogate for driving), and navigated through an alphabetized list of 150 song titles—rendered as an auditory menu—as a secondary task. The menu was presented either in the typical visual-only manner, or enhanced with text-to-speech (TTS), or TTS plus one of three types of additional auditory cues. In Experiment 2, 34 undergraduates conducted the same secondary task while driving in a simulator. In both experiments, performance on both the primary task (success rate of the game or driving performance) and the secondary task (menu search time) was better with the auditory menus than with no sound. Perceived workload scores as well as user preferences favored the enhanced auditory cue types. These results show that adding audio, and enhanced auditory cues in particular, can allow a driver to operate the menus of in-vehicle technologies more efficiently while driving more safely. Results are discussed with multiple resources theory.

Research has focused on music's negative effects on a driver's attention, whereas little research has addressed the possibility of using music to reduce emotional effects on driving. In the present study, we investigate how music can mitigate the degenerated driving performance associated with angry driving. To this end, fifty-three drivers participated in a simulated driving study either with or without induced anger. Three groups of participants with induced anger drove in a simulator while listening to happy or sad instrumental pieces, or without music. In the control group, anger was not induced and they did not listen to music during driving. The results show that participants who listened to either happy or sad music had significantly fewer driving errors than those who did not listen to music. However, no significant differences were found between happy and sad music conditions. Results are discussed with an affect regulation model and future research.

Vehicle area network (VAN) communications and related services are getting more pervasive [1]. However, even though user-centered design has been emphasized, VAN services have often been developed through a technology-driven approach. This paper presents cross-cultural survey results on VAN services in three different countries: Austria, USA, and South Korea. The current research compared the state-of-the-art of drivers' current in-vehicle technology use and investigated their needs and wants for plausible new services in the near future. Further, we validated our next generation in-vehicle interface concepts stemming from our previous participatory design process [2]. Results showed clear differences between Austrians vs. Americans and Koreans. Even though Koreans and Americans in our survey were older than Austrians, they seemed more open-minded to VAN services (e.g., social networks in car, V2V services, in-vehicle agent, etc) in general and rated them more positively. Through these cross-cultural needs analyses of end users, designers and practitioners are expected to gain insights into developing a standardized service across cultures as well as culturally tuned in-vehicle interfaces. Moreover, we hope that this initial international collaboration can serve as a good test bed for future research and hope to expand our consortium with more colleagues in the AutomotiveUI community for further cross-cultural studies.

Research has shown that affective effects on driving performance and safety are as dangerous as (or even more dangerous than) effects of the secondary tasks [11]. There has been some research on the use of speech-based systems for the intervention, but little research on the use of music has attempted to mitigate a driver's affective states while driving. The current paper identifies various taxonomies of the effects of music and explores plausible research variables, considerations, and practical application directions.

Following laws and provisions passed on the national and international level, the most relevant goal of future traffic and vehicular interfaces is to increase road safety. To alleviate the cognitive load associated with the interaction with the variety of emerging information and assistance systems in the car, subliminal stimulation is assumed to be a promising technique. To assess the potential of subliminal cues that could be used as their interaction means in future vehicles, we have organized a workshop within the frame of the automotive user interfaces conference (AutoUI 2011) to discuss this topic in a group of experts. This paper summarizes the findings from that workshop and should give researchers a starting point for their own activities in the field by indicating sort of grand research challenges and most critical issues. In particular, the goal of this summary article is to make this challenging research field more 'tangible' for researchers working in a range of disciplines, such as engineering, neuroscience, computer science, and psychophysiology. While currently discussed in the automotive domain only, the principles, research questions, and findings could immediately (and easily) be transferred to and adopted in other research fields. Interaction based on subliminal techniques can have an impact on society at large, making significant contributions toward a more natural, convenient, and even relaxing future style of interaction with any complex systems.

To cope with the rising volume of information in humancomputer interfaces, explicit and attentive interaction is more and more frequently replaced by implicit means of information exchange, supported by context-and activity-aware systems and applications. The trend of excessive information is, however, still ongoing, calling for further solutions to reduce a persons cognitive load or level of attention. Subliminal interaction techniques are considered a promising approach to deliver information to a person without causing much supplementary workload. This workshop aims at discussing the potential of subliminal perception to improve the information flow for human-computer interaction in the light of the fact that, up to now, the results have been mixed. One group of researchers has provided evidence that subliminal stimulation works, but the other has found that it does not, or even cannot, work. To clarify this issue, experts from various domains attending the workshop will discuss how subliminal effects can be scientifically supported or how a certain claim could be empirically refuted.

Users interact with many electronic devices via menus such as auditory or visual menus. Auditory menus can either complement or replace visual menus. We investigated how advanced auditory cues enhance auditory menus on a smartphone, with tapping, wheeling, and flicking input gestures. The study evaluated a spindex (speech index), in which audio cues inform users where they are in a menu; 122 undergraduates navigated through a menu of 150 songs. Study variables included auditory cue type (text-to-speech alone or TTS plus spindex), visual display mode (on or off), and input gesture (tapping, wheeling, or flicking). Target search time and subjective workload were lower with spindex than without for all input gestures regardless of visual display mode. The spindex condition was rated subjectively higher than plain speech. The effects of input method and display mode on navigation behaviors were analyzed with the two-stage navigation strategy model. Results are discussed in relation to attention theories and in terms of practical applications.

Considerable research has shown that diverse affective (emotional) states influence cognitive processes and performance. To detect a driver's affective states and regulate them may help increase driving performance and safety. There are some populations who are more vulnerable to issues regarding driving, affect, and affect regulation (e.g., novice drivers, young drivers, older drivers, and drivers with TBI (Traumatic Brain Injury)). This paper describes initial findings from multiple participatory design processes, including interviews with 21 young drivers, and focus groups with a TBI driver and two driver rehab specialists. Depending on user groups, there are distinct issues and needs; therefore, differentiated approaches are needed to design an in-vehicle assistive technology system for a specific target user group.

Most emotion detection research starts with a valence dimension---positive and negative states. However, these approaches have not discriminated the effects of distinct emotions of the same valence. Recent psychological findings have proposed that different emotions may have different impacts even though they belong to the same valence. The current study consists of a simulated driving experiment with two induced affective states that are important in driving contexts, to investigate how anger and fear differently influence driving-related risk perception, driving performance, and perceived workload. Twenty four undergraduates drove under three different road conditions with either induced anger or fear. Anger led to more errors than fear, regardless of difficulty level and error type. Also, participants with induced fear reported greater workload than participants with induced anger. Results are discussed in terms of the cognitive appraisal mechanism and design directions for the in-vehicle emotion detection and regulation system.

Following laws and provisions passed on the national and international level, the most relevant goal of future vehicular interfaces is to increase road safety. To alleviate the cognitive load associated with the interaction with the variety of emerging information and assistance systems in the car (and to increase driving performance as well), subliminal persuasion is assumed to be a promising technique to reduce the amount of information the driver must store and recall. Subliminal cues could be provided across appropriate sensory modalities, according to the specific nature of the current task, and corresponding to drivers' cognitive abilities.

Auditory display research for driving has mainly focused on collision warning signals, and recent studies on auditory in-vehicle information presentation have examined only a limited range of tasks (e.g., cell phone operation tasks or verbal tasks such as reading digit strings). The present study used a dual task paradigm to evaluate a plausible scenario in which users navigated a song list. We applied enhanced auditory menu navigation cues, including spearcons (i.e., compressed speech) and a spindex (i.e., a speech index that used brief audio cues to communicate the user's position in a long menu list). Twenty-four undergraduates navigated through an alphabetized song list of 150 song titles---rendered as an auditory menu---while they concurrently played a simple, perceptual-motor, ball-catching game. The menu was presented with text-to-speech (TTS) alone, TTS plus one of three types of enhanced auditory cues, or no sound at all. Both performance of the primary task (success rate of the game) and the secondary task (menu search time) were better with the auditory menus than with no sound. Subjective workload scores (NASA TLX) and user preferences favored the enhanced auditory cue types. Results are discussed in terms of multiple resources theory and practical IVT design applications.

Recently, the use of in-vehicle navigation devices, such as PNDs (Personal or Portable Navigation Devices) has become pervasive, and the device functions have been rapidly expanded and updated. Unfortunately, drivers often have considerable difficulty using these complex technologies. To improve and optimize PND user interfaces, the present study suggested several display improvements for the turning point, which is one of the critical usability issues. Advanced Turn-By-Turn Display and Spatial Turning Sound were suggested to facilitate the preparation of the next turns. Leading Tones for Turning was also presented to help drivers tune the timing of their turns. We evaluated these new concepts with domain experts in three countries, and improved the details of the functions. We are currently implementing those features and looking forward to demonstrating new displays on the real product in our presentation at the Automotive User Interface conference.