Navigating new indoor spaces and interacting with the environment presents many challenges for people who are blind or have low vision (BLV). To address these challenges, we prototyped a smartphone-based conversational assistant that helps BLV people navigate and interact with their environment. The prototype utilizes a cognitive architecture to integrate three different technologies: (i) augmented-reality spatial anchors for high-precision localization and access to static information about the environment; (ii) real-time object/people detection for information about the environment and obstacle avoidance; and (iii) a conversational agent}that uses large language models (LLMs) for information extraction, conversational interaction, and turn-by-turn navigation. We assess the impact of different technologies on human performance by measuring user task time and errors. We found that conversational interaction holistically integrates the different technologies to deliver a better user experience while significantly reducing task completion time.

On-demand, last-mile delivery -- the transportation of goods from a local distribution center to the customer's door within a specified time window -- is used by people with disabilities (PwDs) for a variety of reasons. While delivery robots have potential in this space, they often widen disparities in access for PwDs. Inspired by the concept of pluriversality, which embraces the diversity of worldviews, we facilitated participatory design workshops with PwDs to co-design accessible and equitable delivery robots. Our findings support the importance of delivery robots with varied form factors that are robust and customizable to support diverse PwDs and their respective needs and preferences. We also provide broader considerations for automation in delivery ecosystems.

Guide robots have the potential to improve the experience of independent travel for people who are blind or have low vision. While many technical challenges for robot guides have been studied, open questions about robot behaviors and features remain. We conducted a two-phase user study with 16 blind and low-vision participants. First, we tested whether robot path planners that account for specific orientation cues affect users’ comfort and spatial awareness. Then, we elicited user preferences for guide robots through semi-structured interviews and scenario-based design sessions. We provide insights regarding desired robot behaviors and features that impact robot usability, human agency vs. autonomy, and perceived safety.

Interpersonal synchrony (IS), the behavioral and physiological coordination across time and space, plays a crucial role in social interactions by fostering empathy, closeness, and prosocial behaviors. However, there is a need for more examination of human-robot interaction (HRI) research focused on interactions where the temporal alignment of body movements and the spatial coordination of the content produced by those movements is vital to the quality of the interaction, such as in joint visual art-making. In this work, we investigated the impact of IS on human raters’ perceptions of a human-robot (HR) dyad engaged in a joint painting activity. We conducted two online studies (n = 70, total) in which participants watched 4 videos (1 repeated synchronous video and 3 asynchronous videos). We varied the degree of IS displayed by an HR dyad on two axes: (a) temporal alignment (e.g., speed of producing brush strokes) and (b) spatial similarity (i.e., similarity in the visual content produced). Our results indicate that some temporal and spatial dimensions of IS displayed by an HR dyad during joint painting have significant positive impacts on external observers’ perceptions of the robot, including prosocial tendencies (i.e., empathy, synchrony, and closeness) and acceptance. These findings are significant for emergent research on collaborative robots.

As artificial intelligence (AI) systems have already proven useful in human lives generally, there is an opportunity for specialized human-AI interaction (HAI) systems to support and provide care for older adults with mild cognitive impairment (MCI). However, the integration of this technology in this population must be thoughtfully designed to accommodate specific needs and limitations. This includes careful measurement of both humans and systems. We developed an evolving dataset categorizing relevant measurement tools into five groups: cognitive ability, demographics & personality, activity level, state of mind, and perceptions of the AI system. Each instance of the tool being used in the literature cataloged in the dataset is qualified in terms of how likely we would recommend using it in the domain of HAI for older adults with MCI based on contextual factors and internal reliability measures. This dataset will serve as a valuable resource for future research, aiding in the identification of promising areas and trends in AI systems for older adults with MCI as well as providing essential tools for future studies.

This paper investigates how conversational agents that are embedded in smart environments should present themselves socially. In an online study, we simulate a future space habitat in which "astronauts" (participants) interact with one or more agents to complete several tasks related to science, maintenance, and inventory. We examine effects of agent affiliation (affiliation with a user, affiliation with a domain, or affiliation with all users and all domains) and narrative perspective (first-vs. third-person references to parts of the environment) on mental models of the smart environment as one or multiple entities, trust, performance, and social variables. Our findings suggest that in this type of setting, interacting with a single agent may increase mental demand, and that agents that speak about embodied interaction in third person are perceived as more trustworthy and competent than agents that speak in first person.

With more robots being deployed in the world, users will likely interact with multiple robots sequentially when receiving services. In this paper, we describe an exploratory field study in which unsuspecting participants experienced a "person transfer" – a scenario in which they first interacted with one stationary robot before another mobile robot joined to complete the interaction. In our 7-hour study spanning 4 days, we recorded 18 instances of person transfers with 40+ individuals. We also interviewed 11 participants after the interaction to further understand their experience. We used the recorded video and interview data to extract interesting insights about in-the-field sequential human-robot interaction, such as mobile robot handovers, trust in person transfer, and the importance of the robots’ positions. Our findings expose pitfalls and present important factors to consider when designing sequential human-robot interaction.

When robots have multiple tasks to perform, they must determine the order in which to complete them. Interleaving tasks is efficient for the robot trying to finish its to-do list, but it may be less satisfying for a human whose request was delayed in favor of schedule efficiency. Following online research that examined delays with various motivations, we created two in-person studies in which participants’ tasks were impacted by the robot’s other tasks. In the first, participants either requested a task for the robot to complete on their behalf or watched the robot performing tasks for other people. We measured how their opinions changed depending on whether their task’s completion was delayed due to another participant’s task or they were observing without a task of their own. In the second, participants had a robot walk them to an office and became delayed as the robot detoured to another location. We measured how opinions of the robot changed depending on who requested the detour task and the length of the detour. Overall, participants positively viewed task interleaving as long as the delay and inconvenience imposed by someone else’s task were small and the task was well-justified. Also, observers often had lower opinions of the robot than participants who requested tasks, highlighting a concern for online research.

Many academic subjects are inaccessible for students who are blind or have low vision (BLV) due to the prevalence of visual aids to represent concepts. Interactive devices offer promise as creative tools for teachers of the visually impaired (TVIs) as they can support real-time iteration of adapted learning materials, display changing information, and provide embodied learning experiences for BLV students. We conducted semi-structured interviews with 5 educators (all have TVI experience) and identified their considerations when creating adaptations, attitudes towards technology, and perspectives on existing barriers to access. Our findings reveal and reaffirm unresolved challenges in the adaptation process as well as offer insights into key factors that must be considered when selecting the type of adaptation. From these findings, we formulate design recommendations for interactive tools that support TVIs in creating effective adaptations for BLV students.

Virtual assistants, vacuum robots, security systems, and other smart home technologies are rapidly advancing, evolving, and gaining popularity. This raises questions of how people envision future interactions with smart home systems and how they imagine the future roles of such technologies in society. We deployed an online study that collected fictional short stories from 60 participants about smart home interactions. We identified themes regarding the roles of smart home technologies, social interactions with AI, and concerns about data privacy in the context of the home. We describe our method, discuss insights from the stories that explicitly reflect possible futures and implicitly reflect the present, and make design recommendations based on our findings.

Recent work has suggested that a robot that in-terrupts assigned tasks for the sake of curiosity is perceived as less competent, but that communicating acknowledgment of the curious behavior can mitigate some of those feelings [1]. In real-world situations, there are many reasons why a robot's task could be interrupted in favor of another. For example, a robot handling requests for tasks from people in different locations could navigate more efficiently if it interleaves those tasks, but it ideally would not do so at the expense of the users' perceptions of the robot. In order to understand the impact of different task interleaving patterns on human perceptions of a robot's behavior, we performed a study in which a robot performed a delivery task and an investigative task, interleaving them in various ways. The participants were told either that the investigative task was motivated by a request from another person, motivated by curiosity, or they received no information about why the robot performed the action. While participants acknowledged that interleaving tasks should be allowed, they rated the robot as more competent when its tasks were not interleaved. They were most receptive to interleaving when they knew the investigative task was for another person and less receptive to long task detours away from the delivery route, especially when the inspection task was motivated by curiosity.

Designers of smart-home systems must make decisions about the perceived identities and interconnectedness of their various devices. To inform these decisions, we performed an online study to examine whether people perceive multiple devices in a smart home as different interfaces for the same system, devices that talk to each other, or independent devices. We manipulated the types of devices in the system (hetero-geneous!homogeneous), how the devices relayed commands to each other (implicit/explicit), and the task requested. Participants were flexible in how they interpreted the devices, presenting an opportunity for designers to select a suitable model.

In this paper, we explored how a mobile robot should join an existing multimodal interaction between a person and a stationary robot. We developed three different strategies (Circular, Line, and Improper) and an interactive system to put the strategies in practice. Circular (all interactants stand in a circle) and Line (two robots stand in a line facing a person) were based on existing human group spatial arrangements, whereas Improper (second robot stands far from the person and stationary robot) explored how a bad arrangement might change a user’s position and perception. We also investigated how scenarios with different tasks influenced human positions at different stages of the interaction. We conducted a 3x4 mixed design, in-person user study and found that participants in the Improper condition repositioned themselves to decrease the distance differences between interactants. We also conducted an exploratory analysis on the spatial data to better understand how user actions and social cues, such as gaze and vocalizations, changed spatial behaviors.

As robots are deployed to work in our environments, we must build appropriate expectations of their behavior so that we can trust them to perform their jobs autonomously as we attend to other tasks. Many types of explanations for robot behavior have been proposed, but they have not been fully analyzed for their impact on aligning expectations of robot paths for navigation. In this work, we evaluate several types of robot navigation explanations to understand their impact on the ability of humans to anticipate a robot’s paths. We performed an experiment in which we gave participants an explanation of a robot path and then measured (i) their ability to predict that path, (ii) their allocation of attention on the robot navigating the path versus their own dot-tracking task, and (iii) their subjective ratings of the robot’s predictability and trustworthiness. Our results show that explanations do significantly affect people’s ability to predict robot paths and that explanations that are concise and do not require readers to perform mental transformations are most effective at reducing attention to the robot.

There is considerable effort within the HCI community to explore, document, and advocate for the lived experiences of persons with disabilities (PWDs). However, PWDs from the Global South, particularly Africa, are underrepresented in this scholarship. We contribute to closing this gap by investigating the unmet transit needs and characterization of technology within the disability community in Kampala, Uganda. We investigated transportation due to the increase in ride-share solutions created by widespread mobile computing and the resulting disruption of transportation worldwide. We hosted co-design sessions with disability advocates and adapted the stakeholder tokens method from the value-sensitive design framework to map the stakeholder ecosystem. Our key insight is the identification of a new group of non-traditional core stakeholders who highlight the values of inclusion, mobility, and safety within the ecosystem. Finally, we discuss how our findings engage with concepts of disability justice and perceptions of power.

How are people using current smart home technologies, and how do they conceptualize future ones that are more interconnected and more capable than those available today? We deployed an online survey study to 150 participants to investigate use of and opinions about smart speakers, home robots, virtual assistants, and other smart home devices. We also gauged how impressions of connected smart home devices are shaped by the way the devices interact with one another. Through a mixed-methods qualitative and quantitative approach, we found that people mostly use single devices for single functions, and have simple and brief interactions with virtual assistants. However, they imagine their future devices to have more control over the physical environment (i.e., interact with each other) and envision them interacting with people in more socially complex ways. These findings motivate design considerations and research directions for connected smart home technologies.

We explored different ways in which a multi-robot system might recover after one robot experiences a failure. We compared four recovery conditions: Update (a robot fixes its error and continues the task), Re-embody (a robot transfers its intelligence to a different body), Call (the failed robot summons a second robot to take its place), and Sense (a second robot detects the failure and proactively takes the place of the first robot). We found that trust in the system and perceived competence of the system were higher when a single robot recovered from a failure on its own (by updating or re-embodying) than when a second robot took over the task. We also found evidence that two robots that used the same socially interactive intelligence were perceived more similarly than two robots with different intelligences. Finally, our study revealed a relationship between how people perceive the agency of a robot and how they perceive the performance of the system.

The COVID-19 pandemic has forced institutions to rapidly alter their behavior, which typically has disproportionate negative effects on people with disabilities as accessibility is overlooked. To investigate these issues, we analyzed Twitter data to examine accessibility problems surfaced by the crisis. We identified three key domains at the intersection of accessibility and technology: (i) the allocation of product delivery services, (ii) the transition to remote education, and (iii) the dissemination of public health information. We found that essential retailers expanded their high-risk customer shopping hours and pick-up and delivery services, but individuals with disabilities still lacked necessary access to goods and services. Long-experienced access barriers to online education were exacerbated by the abrupt transition of in-person to remote instruction. Finally, public health messaging has been inconsistent and inaccessible, which is unacceptable during a rapidly-evolving crisis. We argue that organizations should create flexible, accessible technology and policies in calm times to be adaptable in times of crisis to serve individuals with diverse needs.

Smart technologies have recently come under scrutiny for automating inequality. Given the current push towards developing and implementing smart cities policies that affect transportation systems in places like Kampala and Kigali, it is important to examine how the different modes of transportation meet the needs of diverse passengers and identify opportunities for technology to address any inequities. Prior studies have focused on the impact of informal public transportation on government policy and examined drivers' perspectives, but they largely overlooked the experiences of passengers and other industry stakeholders. In this study, we conducted interviews and surveys with public transportation riders with different disabilities as well as other stakeholders, including transport and financial technology creators. Our findings illuminate inequities in the transportation system surrounding discrimination and harassment, influence of ability on preferred transportation modes despite inaccessible interfaces, and influence of perceived social hierarchical structures on innovation. We present insights into how passengers appropriate technology to overcome challenges, and we uncover opportunities for technology to fill additional gaps. Lastly, we discuss how these findings support emergent frameworks such as aspiration-based design, and we present potential envisioned futures of technology for informal public transportation.

People take to social media to share their thoughts, joys, and sorrows. A recent popular trend has been to support and mourn people and pets that have died as well as other objects that have suffered catastrophic damage. As several popular robots have been discontinued, including the Opportunity Rover, Jibo, and Kuri, we are interested in how language used to mourn these robots compares to that to mourn people, animals, and other objects. We performed a study in which we asked participants to categorize deidentified Twitter reactions as referencing the death of a person, an animal, a robot, or another object. Most reactions were labeled as being about humans, which suggests that people use similar language to describe feelings for animate and inanimate entities. We used a natural language toolkit to analyze language from a larger set of tweets. A majority of tweets about Opportunity included second-person ("you") and gendered third-person pronouns (she/he versus it), but terms like "R.I.P" were reserved almost exclusively for humans and animals. Our findings suggest that people verbally mourn robots similarly to living things, but reserve some language for people.

Service robots often perform their main functions in public settings, interacting with more than one person at a time. How these robots should handle the affairs of individual users while also behaving appropriately when others are present is an open question. One option is to design for flexible agent embodiment: letting agents take control of different robots as people move between contexts. Through structured User Enactments, we explored how agents embodied within a single robot might interact with multiple people. Participants interacted with a robot embodied by a singular service agent, agents that re-embody in different robots and devices, and agents that co-embody within the same robot. Findings reveal key insights about the promise of re-embodiment and co-embodiment as design paradigms as well as what people value during interactions with service robots that use personalization.

In robot telemanipulation tasks, the robot can sometimes occlude a target object from the user's view. We investigate the potential of diminished reality to address this problem. Our method uses an optical see-through head-mounted display to create a diminished reality illusion that the robot is transparent, allowing users to see occluded areas behind the robot. To investigate benefits and drawbacks of robot transparency, we conducted a user study that examined diminished reality in a simple telemanipulation task involving both occluded and unoccluded targets. We discovered that while these visualizations show promise for reducing user effort, there are drawbacks in terms of task efficiency and user preference. We identified several friction points in user experiences with diminished reality interfaces. Finally, we describe several design trade-offs among different visualization options.

The inevitable increase in real-world robot applications will, consequently, lead to more opportunities for robots to have observable failures. Although previous work has explored interaction during robot failure and discussed hypothetical danger, little is known about human reactions to actual robot behaviors involving property damage or bodily harm. An additional, largely unexplored complication is the possible influence of social characteristics in robot design. In this work, we sought to explore these issues through an in-person study with a real robot capable of inducing perceived property damage and personal harm. Participants observed a robot packing groceries and had opportunities to react to and assist the robot in multiple failure cases. Prior exposure to damage and threat failures decreased assistance rates from approximately 81% to 60%, with variations due to robot facial expressions and other factors. Qualitative data was then analyzed to identify interaction design needs and opportunities for failing robots.

We describe an exploration of how kiosk-type stationary robots might provide navigation instructions for blind people. Inspired by a technique used by Orientation & Mobility experts in which a route is traced out on a person's palm, we developed five methods that supplement verbal instructions with physical movements. We explored the usability, strengths, and limitations of each of our methods in two exploratory studies with blind participants. One method, in which the robot used its entire arm to create path gestures while participants held its gripper, was preferred by 5 out of 8 blind participants and performed comparably on a recall task as a verbal-only instruction method. A closer approximation of the original palm method failed. We analyzed interview data to understand the reasons behind the failures and successes. We discuss the lessons learned from our studies about instruction methods, how robots in public settings can be useful for blind people, and the challenges of deploying such systems in public.

Virtual Reality (VR) can greatly benefit Human-Robot Interaction (HRI) as a tool to effectively iterate across robot designs. However, possible system limitations of VR could influence the results such that they do not fully reflect real-life encounters with robots. In order to better deploy VR in HRI, we need to establish a basic understanding of what the differences are between HRI studies in the real world and in VR. This paper investigates the differences between the real life and VR with a focus on proxemic preferences, in combination with exploring the effects of visual familiarity and spatial sound within the VR experience. Results suggested that people prefer closer interaction distances with a real, physical robot than with a virtual robot in VR. Additionally, the virtual robot was perceived as more discomforting than the real robot, which could result in the differences in proxemics. Overall, these results indicate that the perception of the robot has to be evaluated before the interaction can be studied. However, the results also suggested that VR settings with different visual familiarities are consistent with each other in how they affect HRI proxemics and virtual robot perceptions, indicating the freedom to study HRI in various scenarios in VR. The effect of spatial sound in VR drew a more complex picture and thus calls for more in-depth research to understand its influence on HRI in VR.

Human-robot interactions that involve multiple robots are becoming common. It is crucial to understand how multiple robots should transfer information and transition users between them. To investigate this, we designed a 3 × 3 mixed-design study in which participants took part in a navigation task. Participants interacted with a stationary robot who summoned a functional (not explicitly social) mobile robot to guide them. Each participant experienced the three types of robot-robot interaction: representative (the stationary robot spoke to the participant on behalf of the mobile robot), direct (the stationary robot delivered the request to the mobile robot in a straightforward manner), and social (the stationary robot delivered the request to the mobile robot in a social manner). Each participant witnessed only one type of robot-robot communication: silent (the robots covertly communicated), explicit (the robots acknowledged that they were communicating), or reciting (the stationary robot said the request aloud). Our results show that it is possible to instill socialness in and improve likability of a functional robot by having a social robot interact socially with it. We also found that covertly exchanging information is less desirable than reciting information aloud.

We explore the effects of robot failure severity (no failure vs. low-impact vs. high-impact) on people's subjective ratings of the robot. We designed an escape room scenario in which one participant teams up with a remotely-controlled Pepper robot. We manipulated the robot's performance at the end of the game: the robot would either correctly follow the participant's instructions (control condition), the robot would fail but people could still complete the task of escaping the room (low-impact condition), or the robot's failure would cause the game to be lost (high-impact condition). Results showed no difference across conditions for people's ratings of the robot in terms of warmth, competence, and discomfort. However, people in the low-impact condition had significantly less faith in the robot's robustness in future escape room scenarios. Open-ended questions revealed interesting trends that are worth pursuing in the future: people may view task performance as a team effort and may blame their team or themselves more for the robot failure in case of a high-impact failure as compared to the low-impact failure.

We explored whether a robot can leverage social influences to motivate nearby bystanders to intervene and defend them from human abuse. We designed a between-subjects study where 48 participants took part in a memorization task and observed a confederate mistreating a robot both verbally and physically. The robot was either empathetic towards the participant»s performance in the task or indifferent. When the robot was mistreated, it ignored the abuse, shut down in response to it, or reacted emotionally. We found that the majority of the participants intervened to help the robot after it was abused. Interventions happened for a wide range of reasons. Interestingly, the empathetic robot increased the proportion of participants that self-reported intervening in comparison to the indifferent robot, but more participants moved the robot as a response to abuse in the latter case. The participants also perceived the robot being verbally mistreated more and reported higher levels of personal distress when the robot briefly shut down after abuse in comparison to when it reacted emotionally or did not react at all.

As children begin to watch more television programming on systems that allow for interaction, such as tablets and videogame systems, there are different opportunities to engage them. For example, the traditional pseudo-interactive features that cue young children's participation in television viewing (e.g., asking a question and pausing for two seconds to allow for an answer) can be restructured to include correct response timing by the program or eventually even feedback. We performed three studies to examine the effects of accurate program response times, repeating unanswered questions, and providing feedback on the children's likelihood of response. We find that three- to five-year-old children are more likely to verbally engage with programs that wait for their response and repeat unanswered questions. However, providing feedback did not affect response rates for children in this age range.

We conducted a 2x2 between-subjects experiment to examine the effects of two orientation and two gaze behaviors during group conversations for a mobile, low degree-of-freedom robot. For this experiment, we designed a novel protocol to induce changes in the robot's group and study different social contexts. In addition, we implemented a perception system to track participants and control the robot's orientation and gaze with little human intervention. The results showed that the gaze behaviors under consideration affected the participants' perception of the robot's motion and that its motion affected human perception of its gaze. This mutual dependency implies that robot gaze and body motion must be designed and controlled jointly, rather than independently of each other. Moreover, the orientation behaviors that we studied led to similar feelings of inclusion and sense of belonging to the robot's group, suggesting that both can be primitives for more complex orientation behaviors.

We designed an augmented reality interface for dialog that enables the control of multimodal behaviors in telepresence robot applications. This interface, when paired with a telepresence robot, enables a single operator to accurately control and coordinate the robot's verbal and nonverbal behaviors. Depending on the complexity of the desired interaction, however, some applications might benefit from having multiple operators control different interaction modalities. As such, our interface can be used by either a single operator or pair of operators. In the paired-operator system, one operator controls verbal behaviors while the other controls nonverbal behaviors. A within-subjects user study was conducted to assess the usefulness and validity of our interface in both single and paired-operator setups. When faced with hard tasks, coordination between verbal and nonverbal behavior improves in the single-operator condition. Despite single operators being slower to produce verbal responses, verbal error rates were unaffected by our conditions. Finally, significantly improved presence measures such as mental immersion, sensory engagement, ability to view and understand the dialog partner, and degree of emotion occur for single operators that control both the verbal and nonverbal behaviors of the robot.

Animated characters are commonly used in children's television, movies, and applications. Artists seek to create characters that maximally engage their audiences and tailor these characters carefully. In order to examine the relationship between stylistic elements of animated characters and the target ages of their audiences, we performed a series of qualitative and quantitative studies. By using existing media, we determined that characters created for younger children have larger head height, larger eye height, and rounder eyes than those created for older children. However, we found no systematic differences by age when we had children express preferences for existing characters or create their own characters. These results suggest that current artistic trends do not accurately reflect the character design preferences of children.

Autism spectrum disorder (ASD) is characterized by unusual social communication and interaction. These traits are often targets for intervention, particularly computer-based interventions (CBIs). We examined whether interactive behaviors in children with autism could be influenced by modifying the facial characteristics of computer avatars and how behavior toward avatars compared to that toward video. Participants spoke with a therapist over a modified videoconferencing system that permitted manipulation of her appearance (i.e., using cartoon or more realistic avatars versus video) and motion (i.e., exaggerating or damping facial movements). We measured the participants' speech, gaze, and gestures. In the first study, we found that the appearance complexity of the avatar did not significantly affect any social interaction behaviors. However, the results of the second study suggest that exaggerated facial motion can improve nonverbal social behaviors, such as gaze and gesture. These findings have implications for character design in CBIs for ASD.

We describe techniques that allow inexpensive, ultra-thin, battery-free Radio Frequency Identification (RFID) tags to be turned into simple paper input devices. We use sensing and signal processing techniques that determine how a tag is being manipulated by the user via an RFID reader and show how tags may be enhanced with a simple set of conductive traces that can be printed on paper, stencil-traced, or even hand-drawn. These traces modify the behavior of contiguous tags to serve as input devices. Our techniques provide the capability to use off-the-shelf RFID tags to sense touch, cover, overlap of tags by conductive or dielectric (insulating) materials, and tag movement trajectories. Paper prototypes can be made functional in seconds. Due to the rapid deployability and low cost of the tags used, we can create a new class of interactive paper devices that are drawn on demand for simple tasks. These capabilities allow new interactive possibilities for pop-up books and other papercraft objects.

Effective teleoperation requires real-time control of a remote robotic system. In this work, we develop a controller for realizing smooth and accurate motion of a robotic head with application to a teleoperation system for the Furhat robot head [1], which we call TeleFurhat. The controller uses the head motion of an operator measured by a Microsoft Kinect 2 sensor as reference and applies a processing framework to condition and render the motion on the robot head. The processing framework includes a pre-filter based on a moving average filter, a neural network-based model for improving the accuracy of the raw pose measurements of Kinect, and a constrained-state Kalman filter that uses a minimum jerk model to smooth motion trajectories and limit the magnitude of changes in position, velocity, and acceleration. Our results demonstrate that the robot can reproduce the human head motion in real time with a latency of approximately 100 to 170 ms while operating within its physical limits. Furthermore, viewers prefer our new method over rendering the raw pose data from Kinect.

Animated characters are expected to fulfill a variety of social roles across different domains. To be successful and effective, these characters must display a wide range of personalities. Designers and animators create characters with appropriate personalities by using their intuition and artistic expertise. Our goal is to provide evidence-based principles for creating social characters. In this article, we describe the results of two experiments that show how exaggerated and damped facial motion magnitude influence impressions of cartoon and more realistic animated characters. In our first experiment, participants watched animated characters that varied in rendering style and facial motion magnitude. The participants then rated the different animated characters on extroversion, warmth, and competence, which are social traits that are relevant for characters used in entertainment, therapy, and education. We found that facial motion magnitude affected these social traits in cartoon and realistic characters differently. Facial motion magnitude affected ratings of cartoon characters’ extroversion and competence more than their warmth. In contrast, facial motion magnitude affected ratings of realistic characters’ extroversion but not their competence nor warmth. We ran a second experiment to extend the results of the first. In the second experiment, we added emotional valence as a variable. We also asked participants to rate the characters on more specific aspects of warmth, such as respectfulness, calmness, and attentiveness. Although the characters’ emotional valence did not affect ratings, we found that facial motion magnitude influenced ratings of the characters’ respectfulness and calmness but not attentiveness. These findings provide a basis for how animators can fine-tune facial motion to control perceptions of animated characters’ personalities.

In recent decades, rates of autism spectrum disorder (ASD) have risen dramatically, and research into assistive technologies for this population has similarly escalated. For technology to be adopted, technologists need to communicate with practitioners across fields and match methodological and evaluation standards. We provide a set of recommendations for researchers to bridge the gap between fields and maximize the impact of their research, including instructions on how to identify and describe research participants and how to avoid research confounds and challenges specific to this population. We also advocate that researchers in ASD maintain a nimble, adaptable approach when performing experiments.

Research indicates that the facial expressions of animated characters and agents can influence people's perceptions and interactions with these entities. We designed an experiment to examine how an interactive animated avatar's facial expressiveness influences dyadic conversations between adults and the avatar. We animated the avatar in realtime using the tracked facial motion of a confederate. To adjust facial expressiveness, we damped and exaggerated the avatar's facial motion. We found that ratings of the avatar's extroversion were positively related to its expressiveness. However, impressions of the avatar's realism and naturalness worsened with increased expressiveness. We also found that the confederate was more influential when she appeared as the damped or exaggerated avatar. Adjusting the expressiveness of interactive animated avatars may be a simple way to influence people's social judgments and willingness to collaborate with animated avatars. These results have implications for using avatar facial expressiveness to improve the effectiveness of avatars in various contexts. Adjusting the expressiveness of interactive animated avatars may be a simple way to influence people's social judgments and willingness to collaborate with animated avatars.

We present a perceptual control space for simulation of cloth that works with any physical simulator, treating it as a black box. The perceptual control space provides intuitive, art-directable control over the simulation behavior based on a learned mapping from common descriptors for cloth ( e.g. , flowiness, softness) to the parameters of the simulation. To learn the mapping, we perform a series of perceptual experiments in which the simulation parameters are varied and participants assess the values of the common terms of the cloth on a scale. A multi-dimensional sub-space regression is performed on the results to build a perceptual generative model over the simulator parameters. We evaluate the perceptual control space by demonstrating that the generative model does in fact create simulated clothing that is rated by participants as having the expected properties. We also show that this perceptual control space generalizes to garments and motions not in the original experiments.

Animated characters appear in applications for entertainment, education, and therapy. When these characters display appropriate emotions for their context, they can be particularly effective. Characters can display emotions by accurately mimicking the facial expressions and vocal cues that people display or by damping or exaggerating the emotionality of the expressions. In this work, we explored which of these strategies would be most effective for animated characters. We investigated the effects of altering the auditory and facial levels of expressiveness on emotion recognition accuracy and ratings of perceived emotional intensity and naturalness. We ran an experiment with emotion (angry, happy, sad), auditory emotion level (low, high), and facial motion magnitude (damped, unaltered, exaggerated) as within-subjects factors. Participants evaluated animations of a character whose facial motion matched that of an actress we tracked using an active appearance model. This method of tracking and animation can capture subtle facial motions in real-time, a necessity for many interactive animated characters. We manipulated auditory emotion level by asking the actress to speak sentences at varying levels, and we manipulated facial motion magnitude by exaggerating and damping the actress's spatial motion. We found that the magnitude of auditory expressiveness was positively related to emotion recognition accuracy and ratings of emotional intensity. The magnitude of facial motion was positively related to ratings of emotional intensity but negatively related to ratings of naturalness.

Interactive animated characters have the potential to engage and educate children, but there is little research on children's interactions with animated characters and real people. We conducted an experiment with 69 children between the ages of 4 and 10 years to investigate how they might engage in conversation differently if their interactive partner appeared as a cartoon character or as a person. A subset of the participants interacted with characters that displayed exaggerated and damped facial motion. The children completed two conversations with an adult confederate who appeared once as herself through video and once as a cartoon character. We measured how much the children spoke and compared their gaze and gesture patterns. We asked them to rate their conversations and indicate their preferred partner. There was no difference in children's conversation behavior with the cartoon character and the person on video, even among those who preferred the person and when the cartoon exhibited altered motion. These results suggest that children will interact with animated characters as they would another person.

For compelling human-robot interaction, social gestures are widely believed to be important. This paper investigates the effects of adding gestures to a physical game between a human and a humanoid robot. Human participants repeatedly threw a ball to the robot, which attempted to catch it. If the catch was successful, the robot threw the ball back to the human. For half of the cases in which the catch was unsuccessful, the robot made a physical gesture, such as shrugging its shoulders, shaking its head, or throwing up its hands. In the other half of cases, no gestures were produced. We used questionnaires and smile detection to compare participants' feelings about the robot when it made gestures after failure versus when it did not. Participants smiled more and rated the robot as more engaging, responsive, and humanlike when it gestured. We conclude that social gesturing of a robot enhances physical interactions between humans and robots.

We propose a method of representing audience behavior through facial and body motions from a single video stream, and use these features to predict the rating for feature-length movies. This is a very challenging problem as: i) the movie viewing environment is dark and contains views of people at different scales and viewpoints; ii) the duration of feature-length movies is long (80-120 mins) so tracking people uninterrupted for this length of time is still an unsolved problem; and iii) expressions and motions of audience members are subtle, short and sparse making labeling of activities unreliable. To circumvent these issues, we use an infrared illuminated test-bed to obtain a visually uniform input. We then utilize motion-history features which capture the subtle movements of a person within a pre-defined volume, and then form a group representation of the audience by a histogram of pair-wise correlations over a small-window of time. Using this group representation, we learn our movie rating classifier from crowd-sourced ratings collected by rottentomatoes.com and show our prediction capability on audiences from 30 movies across 250 subjects (> 50 hrs).

Creating facial animations that convey an animator’s intent is a difficult task because animation techniques are necessarily an approximation of the subtle motion of the face. Some animation techniques may result in linearization of the motion of vertices in space (blendshapes, for example), and other, simpler techniques may result in linearization of the motion in time. In this article, we consider the problem of animating smiles and explore how these simplifications in space and time affect the perceived genuineness of smiles. We create realistic animations of spontaneous and posed smiles from high-resolution motion capture data for two computer-generated characters. The motion capture data is processed to linearize the spatial or temporal properties of the original animation. Through perceptual experiments, we evaluate the genuineness of the resulting smiles. Both space and time impact the perceived genuineness. We also investigate the effect of head motion in the perception of smiles and show similar results for the impact of linearization on animations with and without head motion. Our results indicate that spontaneous smiles are more heavily affected by linearizing the spatial and temporal properties than posed smiles. Moreover, the spontaneous smiles were more affected by temporal linearization than spatial linearization. Our results are in accordance with previous research on linearities in facial animation and allow us to conclude that a model of smiles must include a nonlinear model of velocities.

Animated characters are frequently used in television programs, movies, and video games, but relatively little is known about how their characteristics affect attention and viewer opinions. We used eyetracking and questionnaires to examine the role of visual complexity and animation style on viewing patterns and ratings of video-recorded and animated movie clips. We created videos of an actress performing and describing a series of actions with blocks. Of the videos, one set included regular HD recordings of the actress. The remaining video sets were animated using motion capture data from that actress for three characters: realistic, cartoon, and robot. Increased facial looking time correlated with unpleasantness ratings for individual characters and clips, determining that animation styles have an effect on both viewing patterns and audience members' subjective opinions of characters. In addition, the method described in this paper can expand future research on character animation.

We use a data-driven approach to study both style and abstraction in sketching of a human face. We gather and analyze data from a number of artists as they sketch a human face from a reference photograph. To achieve different levels of abstraction in the sketches, decreasing time limits were imposed -- from four and a half minutes to fifteen seconds. We analyzed the data at two levels: strokes and geometric shape. In each, we create a model that captures both the style of the different artists and the process of abstraction. These models are then used for a portrait sketch synthesis application. Starting from a novel face photograph, we can synthesize a sketch in the various artistic styles and in different levels of abstraction.

Facial animation often falls short in conveying the nuances present in the facial dynamics of humans. In this article, we investigate the subtleties of the spatial and temporal aspects of eye blinks. Conventional methods for eye blink animation generally employ temporally and spatially symmetric sequences; however, naturally occurring blinks in humans show a pronounced asymmetry on both dimensions. We present an analysis of naturally occurring blinks that was performed by tracking data from high-speed video using active appearance models. Based on this analysis, we generate a set of key-frame parameters that closely match naturally occurring blinks. We compare the perceived naturalness of blinks that are animated based on real data to those created using textbook animation curves. The eye blinks are animated on two characters, a photorealistic model and a cartoon model, to determine the influence of character style. We find that the animated blinks generated from the human data model with fully closing eyelids are consistently perceived as more natural than those created using the various types of blink dynamics proposed in animation textbooks.

We consume video content in a multitude of ways, including in movie theaters, on television, on DVDs and Blu-rays, online, on smart phones, and on portable media players. For quality control purposes, it is important to have a uniform viewing experience across these various platforms. In this work, we focus on voice synchronization, an aspect of video quality that is strongly affected by current post-production and transmission practices. We examined the synchronization of an actor's voice and lip movements in two distinct scenarios. First, we simulated the temporal mismatch between the audio and video tracks that can occur during dubbing or during broadcast. Next, we recreated the pitch changes that result from conversions between formats with different frame rates. We show, for the first time, that these audio visual mismatches affect viewer enjoyment. When temporal synchronization is noticeably absent, there is a decrease in the perceived performance quality and the perceived emotional intensity of a performance. For pitch changes, we find that higher pitch voices are not preferred, especially for male actors. Based on our findings, we advise that mismatched audio and video signals negatively affect viewer experience.