For many robotic tasks, particularly those of service robots operating in human environments, the scope of object detection needs is greater than the available data. Either public datasets do not contain the entire set of objects needed for the task, and/or it is a commercial application that cannot use public datasets for training. Instead of hiring people to hand-label more data to support the integration of new objects into robot perception, we propose an interactive training process requiring zero hand labeling. With as little as 4 minutes of interaction with the robot per object, we demonstrate 99% precision and 57% recall in stationary object detection tasks.

The ability to hand objects to users is a key skill for service robots. While the main purpose of a handover action is to successfully transfer an object to the user, it is also relevant that the robot takes the user's preferences into consideration when deciding which handover strategy to use. In a recent online study, we found evidence that suggests two important facts: (1) Users find value in a robot capable of performing handover tasks in more than one manner, and (2) Different users display different preferences on how they would like to be handed a requested object. Therefore, in this work we are proposing a novel system for learning handover preferences. Our system is evaluated in both simulation and on a real robot using computational perception to estimate human activity and environment type.

Service robots are expected to provide assistance to users by performing useful tasks, such as handing over objects upon request. Most robot-human handover studies implicitly assume that the handover action being executed is the same every time. We postulate that for real scenarios, however, a robot should be capable of accomplishing the handover task using multiple styles of handover, selecting the action that will most likely result in the successful execution of the task and the action that best accommodates different user preferences. This paper addresses the human aspect of this theory, investigating: (1) Will users prefer to have the robot execute more than one type of robot-human handovers? and (2) What factors do users take into account to favor one handover over another? In a survey realized with 62 participants from 2 different countries we conclude that not only is having more than one handover action important for the robot, but also identify 2 factors for selecting the best action autonomously.

Spatial affordance can be defined as the functionality a space, or place, lends to human activity. Different places afford different activity possibilities - sleeping is mostly done in the bedroom, and cooking is mostly done in the kitchen. Semantic place labels like kitchen and bedroom, therefore, provide context with which a robot can better infer human activity. Real rooms, however, often defy simple place labels, as they can be multi-purpose, supporting many different types of human activity. The solution is to identify the spatial affordances associated with the current nexus of human activity - a microlevel place labeling. In this paper, we will demonstrate how to estimate these local spatial affordances by integrating a deep learning based place estimator with human pose estimation. The resulting affordances are then used to improve activity recognition using Bayesian belief network.

Comfortable object handover involves searching for a person, identifying them, and then actually handing them an object. It is a goal of many robotics applications, particularly in healthcare. But people can have a wide variety of physical, perceptual, or cognitive limitations. To address these variable patient conditions with an autonomous robot, a robot must adapt to these individual differences. In this work, we first interviewed nurses regarding common issues faced by healthcare providers. Then we designed a system that interfaces with an individual's electronic health record, and adapts its search and handover capabilities to improve the quality of object handover.

Often overlooked in human-robot interaction is the challenge of people detection. For natural interaction, a robot must detect people without waiting for them to face the camera, get far enough away to be fully present, or center themselves fully within the field of view. Furthermore, it must happen without requiring immense amounts of processing that are not practical for real systems. In this work we focus on person detection in a guidance scenario, where occlusion is particularly prevalent. Using a layered approach with depth images, we can substantially improve detection rates under high levels of occlusion, and enable a robot to detect a target that is moving into and out of the field of view.

Effective communication with a mobile robot using speech is a difficult problem even when you can control the auditory scene. Robot self-noise or ego noise, echoes and reverberation, and human interference are all common sources of decreased intelligibility. Moreover, in real-world settings, these problems are routinely aggravated by a variety of sources of background noise. Military scenarios can be punctuated by high decibel noise from materiel and weaponry that would easily overwhelm a robot's normal speaking volume. Moreover, in nonmilitary settings, fans, computers, alarms, and transportation noise can cause enough interference to make a traditional speech interface unusable. This work presents and evaluates a prototype robotic interface that uses perspective taking to estimate the effectiveness of its own speech presentation and takes steps to improve intelligibility for human listeners.

This video submission demonstrates cooperative human-robot firefighting. A human team leader guides the robot to the fire using a combination of speech and gesture.

Human robot interaction presents a unique set of challenges for biometric person identification. During normal interactions between the robot and a user, a tremendous amount of information is available for identification. Our objective is to use this information to identify users quickly and accurately during interactions with a robot. We present our approach for multimodal person identification using Markov logic networks (MLN). We use appearance, clothing, speaker recognition, and face recognition to identify a person during an interaction where they are speaking to the robot. We demonstrate the effectiveness of our approach using sequences of individuals speaking freely on a topic of their choosing.

Robotic sound localization has traditionally been restricted to either on-robot microphone arrays or embedded microphones in aware environments, each of which have limitations due to their static configurations. This work overcomes the static configuration problems by using visual localization to track multiple wireless microphones in the environment with enough accuracy to combine their auditory streams in a traditional localization algorithm. In this manner, microphones can move or be moved about the environment, and still be combined with existing on-robot microphones to extend array baselines and effective listening ranges without having to re-measure inter-microphone distances.

Person identification is the problem of identifying an individual that a computer system is seeing, hearing, etc. Typically this is accomplished using models of the individual. Over time, however, people change. Unless the models stored by the robot change with them, those models will became less and less reliable over time. This work explores automatic updating of person identification models in the domain of speaker recognition. By fusing together tracking and recognition systems from both visual and auditory perceptual modalities, the robot can robustly identify people during continuous interactions and update its models in real-time, improving rates of speaker classification.

Relative positioning of microphones in an array has a significant impact on how well the array can localize sound sources. Two- or three- dimensional localization accuracy of a randomly distributed array will vary widely with respect to the relative position of the sound source. Where a statically located array would be forced to work with its initial configuration, however, a reconfigurable robotic array does not have to remain in its starting position. Given a known region of interest, robots can autonomously optimize their relative configuration to improve localization accuracy where it matters the most. In this work, we propose 2 separate strategies for optimizing such a robotic array. Evaluations are completed first in simulation, and then deployed to a team of real robots.

Creating a human-robot interface is a daunting experience. Capabilities and functionalities of the interface are dependent on the robustness of many different sensor and input modalities. For example, object recognition poses problems for state-of-the-art vision systems. Speech recognition in noisy environments remains problematic for acoustic systems. Natural language understanding and dialog are often limited to specific domains and baffled by ambiguous or novel utterances. Plans based on domain-specific tasks limit the applicability of dialog managers. The types of sensors used limit spatial knowledge and understanding, and constrain cognitive issues, such as perspective-taking.In this research, we are integrating several modalities, such as vision, audition, and natural language understanding to leverage the existing strengths of each modality and overcome individual weaknesses. We are using visual, acoustic, and linguistic inputs in various combinations to solve such problems as the disambiguation of referents (objects in the environment), localization of human speakers, and determination of the source of utterances and appropriateness of responses when humans and robots interact. For this research, we limit our consideration to the interaction of two humans and one robot in a retrieval scenario. This paper will describe the system and integration of the various modules prior to future testing.

Effective communication with a mobile robot using speech is a difficult problem even when you can control the auditory scene. Robot ego-noise, echoes, and human interference are all common sources of decreased intelligibility. In real-world environments, however, these common problems are supplemented with many different types of background noise sources. For instance, military scenarios might be punctuated by high decibel plane noise and bursts from weaponry that mask parts of the speech output from the robot. Even in non-military settings, however, fans, computers, alarms, and transportation noise can cause enough interference that they might render a traditional speech interface unintelligible. In this work, we seek to overcome these problems by applying robotic advantages of sensing and mobility to a text-to-speech interface. Using perspective taking skills to predict how the human user is being affected by new sound sources, a robot can adjust its speaking patterns and/or reposition itself within the environment to limit the negative impact on intelligibility, making a speech interface easier to use.

In this work, we describe an autonomous mobile robotic system for finding and investigating ambient noise sources in the environment. Motivated by the large negative effect of ambient noise sources on robot audition, the long-term goal is to provide awareness of the auditory scene to a robot, so that it may more effectively act to filter out the interference or re-position itself to increase the signal-to-noise ratio. Here, we concentrate on the discovery of new sources of sound through the use of mobility and directed investigation. This is performed in a two-step process. In the first step, a mobile robot first explores the surrounding acoustical environment, creating evidence grid representations to localize the most influential sound sources in the auditory scene. Then in the second step, the robot investigates each potential sound source location in the environment so as to improve the localization result, and identify volume and directionality characteristics of the sound source. Once every source has been investigated, a noise map of the entire auditory scene is created for use by the robot in avoiding areas of loud ambient noise when performing an auditory task.

A mobile robot can be a rather significant source of noise: noisy fans cool onboard computers, motors are spinning, rubber wheels are squeaking against the floor, and mechanical parts are grinding against each other. Despite these noise problems, robots are often suggested as ideal security and surveillance tools, since they can reliably patrol an area, and remove people from harms way. While the arguments for using a robot are persuasive, the use of a noisy robot for these tasks is questionable. Certainly, they might allow a human to be separated from immediate harm, but given the ease with which people and/or machines can detect changes in the ambient noise, a noisy robot might not be very effective at performing its duties. One solution is to make very small, very quiet robots. But such small robots do not usually do very much either, as their sensing and computational power is minimal. An alternative solution is to use a larger robot, but make the robot aware of its own acoustic signature. Combined with knowledge about sound sources and sound flow through the environment, an acoustically aware robot may hide its own acoustic signature in the ambient noise to reduce the risk of being detected.

Auditory perspective taking is the process of imagining the auditory scene from another's place and inferring what that person can (and cannot) hear, as well as how this affects his or her auditory comprehension. With this inferred knowledge, a conversational partner can then adapt his or her vocal presentation to overcome or cope with competing sounds and other auditory challenges to ensure that what is being said can be understood. In this poster, we explore several aspects of auditory perspective taking in the context of a robot speech and listening interface.

Sound source localization on a mobile robot can be a difficult task due to a variety of problems inherent to a real environment, including robot ego-noise, echoes, and the transient nature of ambient noise. As a result, source localization data are often very noisy and unreliable. In this work, we overcome some of these problems by combining the localization evidence over a variety of robot poses using an evidence grid. The result is a representation that localizes the pertinent objects well over time, can be used to filter poor localization results, and may also be useful for global re-localization from sound localization results

We present an approach that uses Q-learning on individual robotic agents, for coordinating a mission-tasked team of robots in a complex scenario. To reduce the size of the state space, actions are grouped into sets of related behaviors called roles and represented as behavioral assemblages. A role is a finite state automata such as Forager, where the behaviors and their sequencing for finding objects, collecting them, and returning them are already encoded and do not have to be re-learned. Each robot starts out with the same set of possible roles to play, the same perceptual hardware for coordination, and no contact other than perception regarding other members of the team. Over the course of training, a team of Q-learning robots will converge to solutions that best the performance of a well-designed handcrafted homogeneous team.

Q-learning has often been used to learn primitive behaviors, or to coordinate a limited set of motor skills. However, the complexity of the algorithm increases exponentially with the number of states the robot can be in and the number of actions that it can take. Therefore, it is natural to try to reduce the number of states and actions in order to improve the efficiency of the algorithm. Robot behaviors and behavioral assemblages provide a good level of abstraction which could be used to speed up robot learning. Instead of coordinating a set of primitives, we use Q-learning to coordinate a set of well tested behavioral assemblages to accomplish a robot mission. The domain for our experiments is a simple intercept mission. This paper also explores the effects of imperfect perceptual algorithms on learning when this approach is used.