As autonomous social robots become more prevalent in home environments, they must decide where to position themselves within many different types of rooms or spaces, balancing accessibility with staying out of the way. This paper presents a machine learning approach to modeling user preferences for robot parking spots in the home using standard 2D occupancy maps. Our method learns spatial patterns from the information available in the occupancy maps and user-annotated floorplans without requiring specialized inputs. We evaluate the approach using floorplan data from 84 users who provided parking spot preferences after living with and evaluating a social robot in their homes for at least two weeks. Our method significantly outperforms a state-of-the-art baseline focused exclusively on avoiding walking paths. We demonstrate how the approach extends to additional map features and share insights about the types of preference patterns learned by the model. This contribution provides a framework that can incorporate new environmental inputs as robot perception capabilities evolve.

For social robots operating in home environments, identifying appropriate parking locations which are "out of the way" is a challenging and multi-faceted problem. This paper proposes a solution to one core aspect of that problem, specifically a model for estimating locations where the robot may block walking paths through narrow spaces. For generality, this model assumes no a priori knowledge about user behaviors or semantic features in the space, and is derived purely from spatial geometry based on a standard 2D occupancy map. An experimental validation based on self-reported parking spot preferences from long-term robot users demonstrates that the proposed model captures 74% of user preferences, outperforming a naive baseline condition in selecting user-preferred parking spots. The proposed method provides a basis for estimating socially-appropriate parking locations for robots operating in the home or other unstructured social spaces and serves as a foundation for developing more sophisticated parking spot preference models in the future.

For social robots to successfully integrate into daily life in home environments, they will need reliable models of the way people perceive and use space in the home. This paper explores the problem of obtaining annotated training data at scale for subjective judgments about spatial locations. Focusing on the use case of identifying good and bad parking spots for a social robot operating in a home environment, two experiments are presented. The first study shows that the presentation of context-rich 3D images to human annotators yields notably different outcomes from those obtained when using 2D robot navigation maps. We attribute the source of these differences to a set of features visible only in the 3D views and introduce a technique for labeling these features on the 2D maps. The second study reveals that using labeled 2D maps produces annotation data very similar to that obtained using 3D images. Since a labeled 2D map can be generated at a fraction of the cost of a full set of 3D views, we recommend this method as a scalable approach to collecting subjective spatial data annotations in everyday environments.

For social robots like Astro which interact with and adapt to the daily movements of users within the home, realistic simulation of human activity is needed for feature development and testing. This paper presents a framework for simulating daily human activity patterns in home environments at scale, supporting manual configurability of different personas or activity patterns, variation of activity timings, and testing on multiple home layouts. We introduce a method for specifying day-to-day variation in schedules and present a bidirectional constraint propagation algorithm for generating schedules from templates. We validate the expressive power of our framework through a use case scenario analysis and demonstrate that our method can be used to generate data closely resembling human behavior from three public datasets and a self-collected dataset. Our contribution supports systematic testing of social robot behaviors at scale, enables procedural generation of synthetic datasets of human movement in different households, and can help minimize bias in training data, leading to more robust and effective robots for home environments.

This paper describes the development of algorithms that decide when to move, where to move, and how to look for people in a home environment. We introduce a design framework as a tool to guide the development of a social robot to proactively be with people for companionship and assistance in the home. Through a series of experiments ranging from simulations to longitudinal A/B studies, we demonstrate how to utilize the design framework to help guide the evaluation and selection of solutions. We deployed our autonomous robot in a long-term in-situ study and found our proposed approach to be more capable of being co-present with its household members compared to a baseline approach. Conducted in an industry setting, our research approach departs from typical academic practices as the motivations are inherently different. We share our perspective on the differences of industry research when developing a social robot as a commercial product.

For practical reasons, most human-robot interaction (HRI) studies focus on short-term interactions between humans and robots. However, such studies do not capture the difficulty of sustaining engagement and interaction quality across long-term interactions. Many real-world robot applications will require repeated interactions and relationship-building over the long term, and personalization and adaptation to users will be necessary to maintain user engagement and to build rapport and trust between the user and the robot. This full-day workshop brings together perspectives from a variety of research areas, including companion robots, elderly care, and educational robots, in order to provide a forum for sharing and discussing innovations, experiences, works-in-progress, and best practices which address the challenges of personalization in long-term HRI.

This study presents a learning-by-imitation technique that learns social robot interaction behaviors from natural human- human interaction data and requires minimum input from a designer. To solve the problem of responding to ambiguous human actions, a novel topic clustering algorithm based on action cooccurrence frequencies is introduced. The system learns human-readable rules that dictate which action the robot should take, based on the most recent human action and the current estimated topic of conversation. The technique is demonstrated in a scenario where the robot learns to play the role of a travel agent. The proposed technique outperformed several baseline techniques in qualitative and quantitative evaluations. It responded more accurately to ambiguous questions and participants found it was easier to understand, provided more information, and required less effort to interact with.

Learning from human interaction data is a promising approach for developing robot interaction logic, but behaviors learned only from offline data simply represent the most frequent interaction patterns in the training data, without any adaptation for individual differences. We developed a robot that incorporates both data-driven and interactive learning. Our robot first learns high-level dialog and spatial behavior patterns from offline examples of human--human interaction. Then, during live interactions, it chooses among appropriate actions according to its curiosity about the customer's expected behavior, continually updating its predictive model to learn and adapt to each individual. In a user study, we found that participants thought the curious robot was significantly more humanlike with respect to repetitiveness and diversity of behavior, more interesting, and better overall in comparison to a non-curious robot.

Interaction Composer, a visual programming environment designed to enable programmers and non-programmers to collaboratively design social human-robot interactions in the form of state-based flows, has been in use at our laboratory for eight years. The system architecture and the design principles behind the framework have been presented in other work, but in this paper we take a case-study approach, examining several actual examples of the use of this toolkit over an eight-year period. We examine the structure and content of interaction flows, identify common design patterns, and discuss elements of the framework which have proven valuable, features which did not solve their intended purposes, and ways that future systems might better address these issues. It is hoped that the insights gained from this study will contribute to the development of more effective and more usable tools and frameworks for interaction design.

The development of an android with convincingly lifelike appearance and behavior has been a long-standing goal in robotics, and recent years have seen great progress in many of the technologies needed to create such androids. However, it is necessary to actually integrate these technologies into a robot system in order to assess the progress that has been made towards this goal and to identify important areas for future work. To this end, we are developing ERICA, an autonomous android system capable of conversational interaction, featuring advanced sensing and speech synthesis technologies, and arguably the most humanlike android built to date. Although the project is ongoing, initial development of the basic android platform has been completed. In this paper we present an overview of the requirements and design of the platform, describe the development process of an interactive application, report on ERICA's first autonomous public demonstration, and discuss the main technical challenges that remain to be addressed in order to create humanlike, autonomous androids.

Recent studies in human-robot interaction (HRI) have investigated ways to harness the power of the crowd for the purpose of creating robot interaction logic through games and teleoperation interfaces. Sensor networks capable of observing human-human interactions in the real world provide a potentially valuable and scalable source of interaction data that can be used for designing robot behavior. To that end, we present here a fully automated method for reproducing observed real-world social interactions with a robot. The proposed method includes techniques for characterizing the speech and locomotion observed in training interactions, using clustering to identify typical behavior elements and identifying spatial formations using established HRI proxemics models. Behavior logic is learned based on discretized actions captured from the sensor data stream, using a naïve Bayesian classifier. Finally, we propose techniques for reproducing robot speech and locomotion behaviors in a robust way, despite the natural variation of human behaviors and the large amount of sensor noise present in speech recognition. We show our technique in use, training a robot to play the role of a shop clerk in a simple camera shop scenario, and we demonstrate through a comparison experiment that our techniques successfully enabled the generation of socially appropriate speech and locomotion behavior. Notably, the performance of our technique in terms of correct behavior selection was found to be higher than the success rate of speech recognition, indicating its robustness to sensor noise.

The use of 3D range sensors for human position tracking has grown in recent years, especially for augmenting robotic sensing for human-robot interaction. However, extrinsic calibration of the relative positions of 3D range sensors is difficult, due to their limited range, narrow field of view, and distortion at large distances. 2D laser range finders have also been used for pedestrian tracking, providing greater accuracy and coverage at the cost of being more expensive and susceptible to occlusion. In this work, we present two novel techniques for calibrating the positions of 3D range sensors based on shared observations of pedestrians. The first technique uses 3D range sensors alone, and the second technique uses 2D and 3D range sensors together, using the high precision and long range of the 2D sensors to complement the short-range but richer sensing of 3D range sensors. We evaluate the accuracy of both automatic calibration techniques, and we furthermore show that the combination of 2D and 3D sensors gives more robust and accurate calibration than when using 3D sensors alone.

Developing interactive behaviors for social robots presents a number of challenges. It is difficult to interpret the meaning of the details of people's behavior, particularly non-verbal behavior like body positioning, but yet a social robot needs to be contingent to such subtle behaviors. It needs to generate utterances and non-verbal behavior with good timing and coordination. The rules for such behavior are often based on implicit knowledge and thus difficult for a designer to describe or program explicitly. We propose to teach such behaviors to a robot with a learning-by-demonstration approach, using recorded human-human interaction data to identify both the behaviors the robot should perform and the social cues it should respond to. In this study, we present a fully unsupervised approach that uses abstraction and clustering to identify behavior elements and joint interaction states, which are used in a variable-length Markov model predictor to generate socially-appropriate behavior commands for a robot. The proposed technique provides encouraging results despite high amounts of sensor noise, especially in speech recognition. We demonstrate our system with a robot in a shopping scenario.

The development of humanlike service robots which interact socially raises a new question: How can we make good interaction content for such robots? Domain experts specializing in the target service have the knowledge for making such content. Yet, while they can easily engage in good face-to-face interactions, we found it difficult for them to prepare conversational content for a robot in written form. Instead, we propose involving experts as teleoperators in an iterative development process in which the expert develops content, teleoperates a robot using that content, and then revises the content based on that interaction. We propose a software system and design guidelines to enable such an iterative design process. To validate these solutions, we conducted a comparison experiment in the field, with elderly volunteer guides teleoperating a robot at a tourist information center in Nara, Japan. The results showed that our system and guidelines enabled domain experts with no robotics background to create better interaction content and conduct better interactions than domain experts without our system.

When robots serve in urban areas such as shopping malls, they will often be required to approach people in order to initiate service. This paper presents a technique for human-robot interaction that enables a robot to approach people who are passing through an environment. For successful approach, our proposed planner first searches for a target person at public distance zones anticipating his/her future position and behavior. It chooses a person who does not seem busy and can be reached from a frontal direction. Once the robot successfully approaches the person within the social distance zone, it identifies the person's reaction and provides a timely response by coordinating its body orientation. The system was tested in a shopping mall and compared with a simple approaching method. The result demonstrates a significant improvement in approaching performance; the simple method was only 35.1% successful, whereas the proposed technique showed a success rate of 55.9%.

The teleoperation of mobile social robots requires operators to understand facial gestures and other nonverbal communication from a person interacting with the robot. It is also critical for the operator to comprehend the surrounding environment in order to facilitate both navigation and human-robot interaction. Allowing the operator to control the robot's gaze direction can help the operator observe a person's nonverbal communication; however, manually actuating a gaze increases the operator's workload and conflicts with the use of the robot's camera for navigation. To address these problems, the authors developed a teleoperation system that combines automatic control of the robot's gaze and a 3-D graphical representation of the surrounding environment, such as location of items and configuration of a shop. A study where a robot plays the role of a shopkeeper was conducted to validate the effectiveness of the proposed gaze-control technique and control interface. It was demonstrated that the combination of automatic gaze control and representations of spatial relationships improved the quality of the robot's interaction with the customer.

This study provides an in-depth analysis and practical solution to the problem of designing and implementing a human-robot team for simple conversational interactions. Models for operation timing, customer satisfaction and customer-robot interaction are presented, based on which a simulation tool is developed to estimate fan-out and robot team performance. Techniques for managing interaction flow and operator task assignment are introduced. In simulation, the effectiveness of different techniques and factors related to team performance are studied. A case study on deploying multiple robots in a shopping mall is then presented to demonstrate the usefulness of our study in helping the design and implementation of social robots in real-world settings.

This paper presents a computational model for side-by-side walking for human-robot interaction (HRI). In this work we address the importance of future motion utility (motion anticipation) of the two walking partners.

Teleoperation of multiple robots by a single operator has been studied extensively for applications such as search and navigation; however, this concept has never been applied to the field of social, conversational robots. In this paper, we explore the unique challenges posed by the remote operation of multiple social robots, where an operator must perform auditory multitasking to assist multiple interactions at once. It describes the general system requirements in four areas: social human-robot interaction (HRI) design, autonomy design, multirobot coordination, and teleoperation interface design. Based on this design framework, we have developed a system in which a single operator can simultaneously control four robots in conversational interactions with users. Key elements of our implementation include a control architecture enabling the scripting of conditional behavior flows for social interaction, a graphical interface enabling an operator to control one robot at a time while monitoring several others in the background, and a technique called “proactive timing control,” which is an automated method for smoothly interleaving the demands of multiple robots for the operator's attention. We also present metrics for describing and predicting robot performance, and we show experimental results demonstrating the effectiveness of our system through simulations and a laboratory experiment based on real-world interactions.

Awareness of time is particularly important for teleoperation of conversational robots, both for controlling the robot and for estimating interaction success, because people have a low tolerance for long pauses in conversation. Findings have shown that people engaged in high-workload tasks tend to underestimate the passage of time. This study confirms that this problem exists for operators controlling a conversational robot, and it investigates mechanisms for improving temporal awareness and task performance while minimizing workload. In a laboratory experiment, two approaches to helping an operator perform various information input tasks were compared: first, assisting temporal awareness by using a clock display, and second, using autonomy to assist one of the operator's time-dependent tasks. Results revealed that assisting the task itself, even without the clock, improved not only task performance but also the operator's temporal awareness. However, results regarding the effect of the clock were ambiguous: it increased workload in general and did not help temporal awareness overall, but it did improve temporal awareness for text entry tasks in particular. As text entry is an important task for teleoperation of social robots, we further investigated the problem of improving temporal awareness during text entry tasks. As the first experiment suggested the effectiveness of a clock, we further validated that the clock is specifically useful to improve temporal awareness. These results showed that the clock did not increase workload for text entry tasks; however, for touch-typing operators, the results suggested that showing a clock after the end of an interaction, rather than continuously throughout the task, could lower the operator's perceived workload.

This study explores the nature of the multi-robot control problem for social robots. It begins by modeling the overall structure of a human-robot team for social interactions, and implements it for specific applications to dialog-based interactions. Operator activity during control of a social robot is studied. Customer satisfaction is proposed as an important metric for evaluating the performance of a human-robot team for social interactions with customers. Based on the modeling, fan-out of a social robot team can be calculated, and the performance of the team is estimated by simulation. A field trial was conducted in a shopping mall to demonstrate a successful deployment of social robots for a real-world application with ensured performance prior to installation using our modeling and simulation approach.

To provide personal and location-dependent services in public spaces such as shopping malls, it is important to be able to estimate the positions and identities of people in the environment. Sensors in the environment reliably detect their current positions, but it is difficult to identify people using these sensors. On the other hand, signals from wearable sensors can be used to identify people correctly, but precise position estimation remains problematic. In this paper, we describe a novel method of integrating laser range finders (LRFs) in the environment and wearable inertial sensors. Time sequences of angular velocities estimated from both LRFs and wearable sensors are matched to identify people. Examples of tracking individuals in the environment that confirm the effectiveness of this method are shown.

Laser range finders are a non-invasive tool which can be used for anonymously tracking the motion of people and robots in real-world environments with high accuracy. Based on a commercial system we have developed, this paper addresses two practical issues of using networks of portable laser range finders in field environments. We first describe a technique for automated calibration of sensor positions and orientations, by using velocity-based matching of observed human trajectories to define constraints between the sensors. We then propose a mechanism for detecting when a sensor has been moved out of alignment, which can be used to alert an operator of the condition and automatically exclude erroneous data from tracking calculations. After describing our techniques for solving these problems, we demonstrate the effectiveness of our calibration and error detection systems in live trials with our real-time system, as well as offline tests based on scan data recorded from field trials.

Simultaneous teleoperation of mobile, social robots presents unique challenges, combining the real-time demands of conversation with the prioritized scheduling of navigational tasks. We have developed a system in which a single operator can effectively control four mobile robots performing both conversation and navigation. We compare the teleoperation requirements for mobile, social robots with those of traditional robot systems, and we identify metrics for evaluating task difficulty and operator performance for teleoperation of mobile social robots. As a proof of concept, we present an integrated priority model combining real-time conversational demands and non-real-time navigational demands for operator attention, and in a pioneering study, we apply the model and metrics in a demonstration of our multi-robot system through real-world field trials in a shopping arcade.

This paper proposes a model of approach behavior with which a robot can initiate conversation with people who are walking. We developed the model by learning from the failures in a simplistic approach behavior used in a real shopping mall. Sometimes people were unaware of the robot's presence, even when it spoke to them. Sometimes, people were not sure whether the robot was really trying to start a conversation, and they did not start talking with it even though they displayed interest. To prevent such failures, our model includes the following functions: predicting the walking behavior of people, choosing a target person, planning its approaching path, and nonverbally indicating its intention to initiate a conversation. The approach model was implemented and used in a real shopping mall. The field trial demonstrated that our model significantly improves the robot's performance in initiating conversations.

Robust localization of robots and reliable tracking of people are both critical requirements for the deployment of service robots in real-world environments. In crowded public spaces, occlusions can impede localization using on-board sensors. At the same time, teams of service robots working together need to share the locations of people and other robots on the same global coordinate system in order to provide services efficiently. To solve this problem, our approach is to use an infrastructure of sensors embedded in the environment to provide an inertial reference frame and wide-area coverage. Based on a people-tracking system we have previously established which uses laser range finders to track people's trajectories, we have developed a technique to localize a team of service robots on a shared global coordinate system. Each robot's odometry data is associated with the observed trajectory of an entity detected by the laser tracking system, and Kalman filters are used to correct rotational offsets between the robots' individual coordinate systems and the global reference frame. We present our data association and pose correction algorithms and show results demonstrating the performance of our system in a shopping arcade.

This paper reports the challenges of developing multiple social robots that operate in a shopping mall. We developed a networked robot system that coordinates multiple social robots and sensors to provide efficient service to customers. It directs the tasks of robots based on their positions and people's walking behavior, manages the paths of robots, and coordinates the conversation-performance between two robots. Laser range finders were distributed in the environment to estimate people's positions. The system estimates such human walking behaviors as ¿stopping¿ or ¿idle walking¿ to direct robots to provide appropriate tasks to appropriate people. Each robot interacts with people to provide recommendation information and route information about shops. The system sometimes simultaneously uses two robots to lead people from one place to another. The field trial, which was conducted in a shopping mall where four robots interacted with 414 people, revealed the effectiveness of the network robot system for guiding people around a shopping mall as well as increasing their interest.

For a robot providing services to people in a public space such as a shopping mall, it is important to distinguish potential customers, such as window shoppers, from other people, such as busy commuters. In this paper, we present a series of abstraction techniques for people's trajectories and a service framework for using these techniques in a social robot, which enables a designer to make the robot proactively approach customers by only providing information about target local behavior. We placed a ubiquitous sensor network consisting of six laser range finders in a shopping arcade. The system tracks people's positions as well as their local behaviors, such as fast walking, idle walking, wandering, or stopping. We accumulated people's trajectories for a week, applying a clustering technique to the accumulated trajectories to extract information about the use of space and people's typical global behaviors. This information enables the robot to target its services to people who are walking idly or stopping. The robot anticipates both the areas in which people are likely to perform these behaviors as well as the probable local behaviors of individuals a few seconds in the future. In a field experiment, we demonstrate that this service framework enables the robot to serve people efficiently.

Teleoperation of multiple robots has been studied extensively for applications such as robot navigation; however, this concept has never been applied to the field of social robots. To explore the unique challenges posed by the remote operation of multiple social robots, we have implemented a system in which a single operator simultaneously controls up to four robots, all engaging in communication interactions with users. We present a user inter-face designed for operating a single robot while monitoring several others in the background, then we propose methods for characterizing task difficulty and introduce a technique for improving multiple-robot performance by reducing the number of conflicts between robots demanding the operator's attention. Finally, we demonstrate the success of our system in laboratory trials based on real-world interactions.

For a robot providing services to people in a public space such as a train station or a shopping mall, it is important to distinguish potential customers, such as window-shoppers, from other people, such as busy commuters. In this paper, we present a series of techniques for anticipating people's behavior in a public space, mainly based on the analysis of accumulated trajectories, and we demonstrate the use of these techniques in a social robot. We placed a ubiquitous sensor network consisting of six laser range finders in a shopping arcade. The system tracks people's positions as well as their local behaviors such as fast walking, idle walking, or stopping. We accumulated people's trajectories for a week, applying a clustering technique to the accumulated trajectories to extract information about the use of space and people's typical global behaviors. This information enables the robot to target its services to people who are walking idly or stopping. The robot anticipates both the areas in which people are likely to perform these behaviors, and also the probable local behaviors of individuals a few seconds in the future. In a field experiment we demonstrate that this system enables the robot to serve people efficiently.

We have developed a new communication robot, Robopal, which is an indoor/outdoor robot for use in human-robot interaction research in the context of daily life. Robopal's intended applications involve leading and/or following a human to a destination. Preliminary experiments have been conducted to study nonverbal cues associated with leading and following behavior, and it has been observed that some behaviors, such as glancing towards the leader or follower, appear to be role-dependent. A system for representing these behaviors with a state transition model is described, based on four kinds of interaction roles: directive, responsive, collaborative, and independent. It is proposed that behavior modeling can be simplified by using this system to represent changes in the roles the robot and human play in an interaction, and by associating appropriate behaviors to each role

In this paper we present a method for determining body orientation and pose information from laser scanner data using particle filtering with an adaptive modeling algorithm. A parametric human shape model is recursively updated to fit observed data after each resampling step of the particle filter. This updated model is then used in the likelihood estimation step for the following iteration. This method has been implemented and tested by using a network of laser range finders to observe human subjects in a variety of interactions. We present results illustrating that our method can closely track torso and arm movements even with noisy and incomplete sensor data, and we show examples of body language primitives that can be observed from this orientation and positioning information.

Article mediaBlocks: physical containers, transports, and controls for online media Share on Authors: Brygg Ullmer Massachusetts Institute of Technology Massachusetts Institute of TechnologyView Profile , Hiroshi Ishii Massachusetts Institute of Technology Massachusetts Institute of TechnologyView Profile , Dylan Glas Massachusetts Institute of Technology Massachusetts Institute of TechnologyView Profile Authors Info & Claims SIGGRAPH '98: Proceedings of the 25th annual conference on Computer graphics and interactive techniquesJuly 1998 Pages 379–386https://doi.org/10.1145/280814.280940Online:24 July 1998Publication History 138citation1,709DownloadsMetricsTotal Citations138Total Downloads1,709Last 12 Months36Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access