Interaction understanding and activity recognition require the ability to identify patterns in sequences of events. A robot recording what is going on in real-life settings could be able to exploit these patterns using a representation similar to a map of the observed interactions. This paper examines the use of perceptual constructs, generated by deep neural networks that process visual and audio data, to build graph-based interaction mapping from the observation of human activities. Patterns are detected using loop closure detections, similarly to what is done in simultaneous localization and mapping approach. Results suggest that the graph-based interaction mapping approach is able to create condensed representations of interaction events, and find patterns in sequences of perceptual constructs that create loops. Such technique could help design socially intelligent robots that can derive an understanding of their environment from observing and learning from their interactions with people.

In recent years, studies on Socially Assistive Robots (SARs) examine how to improve the quality of life of people living with dementia and older adults (OAs) in general. However, most SARs have somewhat limited perception capabilities or interact using simple pre-programmed responses, providing limited or repetitive interaction modalities. Integrating more advanced perceptual capabilities with deep learning processing would help move beyond such limitations. This paper presents T-Top, a tabletop robot designed with advanced audio and vision processing using deep learning neural networks. T-Top is made available as an open source platform with the goal of providing an experimental SAR platform that can implement richer interaction modalities with OAs.

Socially Assistive Robots are foreseen as having the potential to improve the quality of life of older adults and individuals with mental disabilities. Natural human-robot interaction in everyday settings may require robots that are capable of understanding what is happening in their operating environments so that they can respond appropriately to the experienced situations and engage people in meaningful ways. This paper presents an approach using perceptual constructs to represent what is being observed by the robot. Perceptual constructs are derived from deep neural networks used to process visual and audio data. The objective is to derive a compressed representation of the interactions observed by the robot in real-life settings. Results are provided from observations made by a robot of a room with human activity over a two-week period, outlining what works and remaining challenges.

During these past years, Socially Assistive Robots (SARs) have been used to study the benefits of their uses with elderly people and people with dementia for healthcare purposes. Yet, almost all SARs have somewhat limited perception capabilities or respond using simple pre-programmed behaviors and reactions, providing limited or repetitive interaction modalities. To overcome these limitations and take into consideration the strengths and weaknesses of SARs in healthcare settings, this paper presents T-Top, a tabletop robot designed with advanced audio and vision sensors, deep learning perceptual processing and telecommunication capabilities. Designed as a open hard-ware/software platform, the objective with T-Top is to provide an experimental platform that can implement richer interaction modalities and develop higher cognitive abilities from interacting with people.

A microphone array can provide a mobile robot with the capability of localizing, tracking and separating distant sound sources in 2D, i.e., estimating their relative elevation and azimuth. To combine acoustic data with visual information in real world settings, spatial correlation must be established. The approach explored in this paper consists of having two robots, each equipped with a microphone array, localizing themselves in a shared reference map using SLAM. Based on their locations, data from the microphone arrays are used to triangulate in 3D the location of a sound source in relation to the same map. This strategy results in a novel cooperative sound mapping approach using mobile microphone arrays. Trials are conducted using two mobile robots localizing a static or a moving sound source to examine in which conditions this is possible. Results suggest that errors under 0.3 m are observed when the relative angle between the two robots are above 30° for a static sound source, while errors under 0.3 m for angles between 40° and 140° are observed with a moving sound source.

Rotary-Wing Air Vehicles (RW-UAVs), also referred to as drones, have gained in popularity over the last few years. Intrusions over secured areas have become common and authorities are actively looking for solutions to detect and localize undesired drones. The sound generated by the propellers of the RW-UAVs is powerful enough to be perceived by a human observer nearby. In this paper, we examine the use of particle filtering to detect and localize in 3D the position of a RW-UAV based on sound source localization (SSL) over distributed microphone arrays (MAs). Results show that the proposed method is able to detect and track a drone with precision, as long as the noise emitted by the RW-UAVs dominates the background noise.

Commercial telepresence robots provide video, audio, and proximity data to remote operators through a teleoperation user interface running on standard computing devices. As new modalities such as force sensing and sound localization are being developed and tested on advanced robotic platforms, ways to integrate such information on a teleoperation interface are required. This paper demonstrates the use of visual representations of forces and sound localization in a 3D teleoperation interface. Forces are represented using colors, size, bar graphs and arrows, while speech or ring bubbles are used to represents sound positions and types. Validation of these modalities is done with 31 participants using IRL-1/TR, a humanoid platform equipped with differential elastic actuators to provide compliance and force control of its arms and capable of sound source localization. Results suggest that visual representations of interaction force and sound source can provide appropriately useful information to remote operators.

Mobile devices such as smartphones and tablets can provide additional sensing and interacting capabilities to a mobile robot, even extending its senses to remote locations. To do so, we developed ROS4iOS, a native port of ROS allowing to seamlessly use data from mobile iOS devices to be processed on a robot. To demonstrate this capability, this video presentation illustrates how ROS4iOS has been used to implement an assistance scenario: a person in a remote location asks our IRL-1 robot for assistance, and IRL-1 must recognize the person's voice, identify the remote location using images taken from the mobile device, navigate to the identified location and interact vocally with the person through the mobile device. When communication with the robot is established by launching a specific iOS application, audio from the mobile device is published on a single topic and directed toward two ROS nodes on IRL-1. PocketSphinx, a speech recognition toolkit, is used to obtain the person vocal commands. WISS, a speaker identification system, is used for speaker identification. From a vocal request made through the mobile device, the robot can identify the person and request to get images of the remote location using the rear-facing camera of the mobile device. RTABMap, a loop-closure detection system for visual location recognition, is used to locate the person. A map was previously built with the robot's laser range finder using the gmapping SLAM algorithm, thus permitting reuse of the ROS navigation stack to plan a path and follow it safely from its current location to the requested one. Depending on its current state, the robot can also display the most suited view (i.e., phone, camera, navigation) on the iOS application. ROS4iOS can also be used to teleoperate IRL-1 through the mobile device or to discuss remotely with the robot, using the same ROS modules. ROS4iOS opens up a rich set of possibilities for HRI, making ROS-compatible code accessible on mobile devices. In future work, we plan to integrate a dialog management system for other assistance scenarios, such as image-based object fetching and delivery.

Smartphones and tablets are now part of our everyday lives, facilitating access to information through intuitive graphical interfaces. They can make great devices to work with robots because they integrate a lot of sensors and reasonable computing power. Using ROS on those devices would facilitate code reuse and integration with existing robotics applications and libraries.

Designing robots that interact naturally with people requires the integration of technologies and algorithms for communication modalities such as gestures, movement, facial expressions and user interfaces. To understand interdependence among these modalities, evaluating the integrated design in feasibility studies provides insights about key considerations regarding the robot and potential interaction scenarios, allowing the design to be iteratively refined before larger-scale experiments are planned and conducted. This paper presents three feasibility studies with IRL-1, a new humanoid robot integrating compliant actuators for motion and manipulation along with artificial audition, vision, and facial expressions. These studies explore distinctive capabilities of IRL-1, including the ability to be physically guided by perceiving forces through elastic actuators used for active steering of the omnidirectional platform; the integration of vision, motion and audition for an augmented telepresence interface; and the influence of delays in responding to sounds. In addition to demonstrating how these capabilities can be exploited in human-robot interaction, this paper illustrates intrinsic interrelations between design and evaluation of IRL-1, such as the influence of the contact point in physically guiding the platform, the synchronization between sensory and robot representations in the graphical display, and facial gestures for responsiveness when computationally expensive processes are used. It also outlines ideas regarding more advanced experiments that could be conducted with the platform.

The design and the evaluation of an autonomous interactive robot is a challenging research endeavor because there is as much to learn from the interaction between the integrated technologies as there is from the embodied human-robot interaction, in addition to observing their mutual interdependencies. This paper reports on IRL-0, a prototyping platform that we used to conduct preliminary studies on the influences of combining verbal and nonverbal modalities (facial expressions, head movement, arm gestures, and approach trajectory) for engaging interaction with people in controlled conditions and in real-world settings. IRL-0 is made of a compliant omnidirectional mobile base equipped with an expressive face and a three degrees-of-freedom (DOFs) compliant arm. By assembling this prototype and conducting these preliminary studies, our objective is to acquire insights in terms of design (e.g., technology, control) and experimental procedures that are important to take into consideration for the designing and evaluating autonomous robots engaging interaction with people.

Johnny-0, shown in Figure 1, is our new humanoid robot which integrates an expressive face on an orientable head, two arms with 4 degrees of freedom (DOF) each and grippers, mounted on an omnidirectional, non-holonomic mobile platform. Our underlying goal with Johnny-0 is to design a platform capable of natural reciprocal interaction (motion, language, touch, affect) with humans, to address integration issues associated with advanced motion, interaction and cognition capabilities on the same platform, and their use in unconstrained real world conditions. To do so, compliance is a necessity to provide natural and safe interactions.

To demonstrate the influence of an artificial audition system on speech recognition and dialogue management for a robot, this paper presents a case study involving soft coupling of ManyEars, a sound source localization, tracking and separation system, with the CSLU Dialogue Management system. Trials were conducted in a laboratory and a cafeteria. Results indicate that preprocessing of the audio signals by ManyEars improves speech recognition and dialogue management of the system, demonstrating the feasibility and the added flexibility provided by ManyEars for a robot to interact vocally with humans in a wide variety of contexts.

AZIMUT-3 is an omnidirectional non-holonomic (or pseudo-omnidirectional) robotic platform intended for safe human-robot interaction. In its wheeled configuration, shown in Fig. 1, AZIMUT-3 uses eight actuators for locomotion: four for propulsion and four for steering the wheels, which can rotate 180 degrees around their steering axis. Propulsion is done using standard DC brushless motors (Bayside K064050-3Y) with optical encoders (US Digital E4-300-157-HUB, 0.3 deg of resolution), capable of reaching 1.47 m/s. The platform uses steerable wheels motorized using differential elastic actuators (DEA), which provide compliance, safety and torque control capabilities. AZIMUT-3's hardware architecture consists of distributed modules for sensing and low-level control, communicating with each other through a 1 Mbps CAN bus. A Mini-ITX computer equipped with a 2.0 GHz Core 2 duo processor running Linux with real-time patches (RT-PREEMPT) is used on-board for high-level control modules. Nickel-metal hydride batteries provide power to the platform for up to 3 hours of autonomy. A passive vertical suspension mechanism (Rosta springs) is used to connect the wheels to AZIMUT-3's chassis, allowing them to keep contact with the ground on uneven surfaces. The platform has a 34 kg payload capacity and weights 35 kg.

Robots are usually built using stiff actuators that can provide impressive motion performances. However, they struggle to control the force, they do not handle collisions graciously and are generally bad at interacting significantly with partially unknown or kinematically constrained environments. One solution is to add a force sensor in the closed-loop control of backdrivable actuators [1], but this is limited in terms of stability, safety and robustness [2] [3]. One of the foremost initiatives using this method was undertaken by the German Aerospace Center (DLR) and resulted in three generations of extensively optimized lightweight robotic arms [4] that can physically interact with people. Performances are impressive but robustness is still an issue.

Most modern robotic systems are fast and repeatable position controlled machines. However, because of their inability to interact safely, robustly and in a versatile manner, they mostly remain confined to controlled areas where they execute specific preprogrammed actions. Providing high performance motion simultaneously with the ability to physically interact significantly remains a challenge.

This video shows a light hearted view of a rolling autonomous robot named Roball. Roball is shown interacting with various children who age from 10 months old to teenagers at a high school. The clips show the different ways children interact with Roball and also the different types of reactions the children can have to Roball. Each clip was taken from a trial that was conducted to investigate Child-Robot Interaction (CRI).

There is great potential for robotic devices when being applied with children. They can be used from play to assistive applications. We develop robotic devices for a diverse range of children that differ in age, gender and ability, which includes children that are diagnosed with cognitive difficulties such as autism. Every child is an individual and they vary in their personalities and styles of interaction. Therefore, being able to adjust the robot's behaviour to the type of interaction it is receiving was believed to be essential. In this abstract we examine a series of trials which investigated how adaptation (through changes in motion and sound) on a fully autonomous rolling robot could help gain and sustain the interest of five different children. We discovered surprising benefits to having adaptation on-board Roball.

Auditory capabilities would allow small robots interacting with people to act according to vocal cues. In our recent work, we have demonstrated AUDIBLE, an auditory system capable of sound source localization, tracking and separation in real-time, using an array of eight microphones and running on a laptop computer. The system is able to localize and track up to four sources, while separating up to three sources in real-time in noisy environments. Signal processing techniques can be quite computer intensive, and the question of making it possible for this system to run on platforms that cannot carry a laptop computer onboard can be raised. This paper reports our investigation of the appropriate compromises to be made to AUDIBLE's implementation in order to port the system on an embedded DSP (Digital Signal Processor) platform. The DSP implementation is fully functional and performs well with minor limitations compared to the original system i.e., limitations on sound source duration and on the number of sources that can be processed simultaneously. Results demonstrate that it is feasible to port AUDIBLE on embedded platforms, opening up its use in field applications such as human-robot interaction in real life settings.

Coordination of a group of mobile robots is facilitated when they are able to determine their positions relative to each other. Instead of using an absolute positioning approach with fixed beacons in the operating environment, we have developed a ultrasonic relative positioning system that allows each robot to perceive the distance and the angle of other nearby robots in relation to its own position. The system is based on time-of-flight evaluation of ultrasonic pulses and a RF communication link. The system has a precision of 8 mm and of 3deg over a 6.7 m range. This paper describes the system, its performance and its use on four Pioneer 2 robots moving in formation.

Behavior-based architectures use behaviors as building blocks for decision-making and action execution processes. Behaviors are distributed and evaluated in parallel for the control of the robot, taking real-time inputs from sensory data and sending real-time commands to effectors. No centralized components exist in these architectures, each module carrying out its own strategy independently, making an overall behavior emerge from the interaction between the concurrently executed modules and the environment. In this paper, we discuss the use of a reactive hierarchical task network (HTN) planner in a behavior-based robot architecture. The planner in this architecture is not a central component on which everything else relies on, but acts as one of the motivational modules recommending tasks to be executed and influencing the selection and configuration of behaviors. The planning module allows the behavior-based architecture to deal with tasks with priorities, flexible time constraints and on-line planning using a simple but very effective reactive planning strategy. We demonstrate our approach in the context of making a robot attend a conference.

The most common ground locomotion method to make a mobile robot move is to use two-wheel drive with differential steering and a rear balancing caster. Controlling the two motors independently makes the robot non-holonomic in its motion. Such robots can work well indoor on flat surfaces and in environments adapted for wheelchairs. But the benefit of providing mobility to a robot directly relies on its locomotion capability, for handling different types of terrains (indoors or outdoors) and situations such as moving slowly or rapidly, with or without the presence of moving objects (living or not), climbing over objects and potentially having to deal with hazardous conditions. It is with this objective in mind that we designed AZIMUT. AZIMUT is a legged tracked wheeled robot capable of changing the orientation of its four articulations. Each articulation has three degrees of freedom (DOF): it can rotate 360deg around its point of attachment to the chassis, can change its orientation over 180deg, and rotate to propulse the robot.

This video shows the functionalities of a 3 serial DOF robotic arm. Each DOF is actuated with a patent pending differential elastic actuator (DEA) [1,2]. Compared to the abundantly studied series elastic actuator [3,4], DEA uses a differential coupling between a high impedance mechanical speed source and a low impedance mechanical spring. Possible implementations of a mechanical differential include the use of a standard gearbox, harmonic drive, cycloidal gearbox, bar mechanism, cable mechanism and all other mechanism that implement a differential function between three mechanical ports. For the implementation reported in this video, we used a harmonic drive for a very compact design. A passive torsion spring (thus the name elastic), with a known impedance characteristic corresponding to the spring stiffness, is used, with an electrical DC brushless motor. A non-turning sensor connected in series with the spring measures the torque output of the actuator.

Increasingly researchers are looking outside of normal communication channels (such as video and audio) to provide additional forms of communication or interaction between a human and a robot, or a robot and its environment. Amongst the new channels being investigated is the detection of touch using infrared, proprioceptive and temperature sensors. Our work aims at developing a system that can detect natural touch or interaction coming from children playing with a robot, and adapt to this interaction. This paper reports trials carried out using Roball, a spherical mobile robot, demonstrating how sensory data patterns can be identified in human-robot interaction, and exploited for achieving behavioral adaptation. The experimental methodology used for these trials is reported, which validated the hypothesis that human interaction can not only be perceived from proprioceptive sensors on-board a robotic platform, but that this perception has the ability to lead to adaptation.

To eventually have automated vehicles operate in platoons, it is necessary to study what information each vehicle must have and to whom it must communicate for safe and efficient maneuvering in all possible conditions. This paper formulates the problem in terms of sensing and communicated information. By emulating platoons using a group of mobile robots, the authors demonstrate the feasibility of maneuvers (such as entering, exiting, and recuperating from an accident) using different distributed coordination strategies. The coordination strategies studied range from no communication to unidirectional or bidirectional exchanges between vehicles and to fully centralized decision by the leading vehicle. One particularity of this paper is that instead of assuming that the platoon leader or all vehicles globally monitor what is going on, only the vehicles involved in a particular maneuver are concerned, distributing decisions locally among the platoon. This paper reports experimental trials using robots having limited and directional perception of other things, using vision and obstacle avoidance sensing. Results confirm the feasibility of the coordination strategies in different conditions and various uses of communicated information to compensate for sensing limitations

This paper presents the design process of a spherical robot capable of autonomous motion, and demonstrates how it can become a tool in child-development studies. The robot, named Roball, is capable of intentional self-propelled movements and can generate various interplay situations using motion, messages, sounds, illuminated parts and other sensors. Such capabilities allow Roball to interact with young children in simple and interesting ways, and to provide the potential of contributing to the development of their language, affective, motor, intellectual and social skills. Trials done with 12-24-month-old children demonstrate how Roball can be used to study children's interest in a self-propelled and intentional device. An experimental methodology to conduct such studies is presented: it is based on quantitative and qualitative techniques to evaluate interactions, thus enabling the identification of challenges and opportunities in child-robot interaction studies.

Real life deployment of robot formation cannot assume that robots are going to be correctly positioned to move in a particular configuration. To do so, we propose an approach that allows the group to determine autonomously the most appropriate assignment of positions in the formation. Our approach is distributed and uses directional visual perception to localize robots. Inter-robot communication allows them to share information on which robots are nearby, so that each can evaluate it ability to be the conductor of the group and assign formation positions to the other robots by minimizing repositioning. The assignment search is done using a distributed bounded depth-first with pruning search. The robot with the best score is selected as the conductor, and the other robots receive from the conductor their assignment in the formation. Validation of our work is done in simulation and with Pioneer 2 robots.

This paper describes two initiatives aiming at improving code reusability for programming mobile robots: robotflow/flowdesigner, a data-flow programming environment; MARIE (mobile and autonomous robotics integration environment), a programming environment allowing multiple applications, programs and tools, to operate on one or multiple machines/OS and work together on a mobile robot implementation. Robotflow/flowdesigner's objective is to provide a modular, graphical programming environment that would help visualize and understand what is really happening in the robot's control loops, sensors, actuators, by using graphical probes. MARIE aims at avoiding making an exclusive choice on particular programming tools, making it possible to reuse code and applications.

For robots to move in formation or to make them cooperate for distributed sensing of an area, they need to be able to discriminate their position relative to each other. Instead of using an absolute positioning approach, we have developed a relative positioning device that allows to perceive the distance and the angle of other nearby devices. The system is based on time-of-flight evaluation of ultrasonic waves and a RF communication link. The approach is validated using two Pioneer 2 robots in a leader-follower configuration.

Giving the ability to read characters and symbols is highly desirable for increased autonomy of mobile robots operating in the real world. The idea is fairly simple: give a robot the ability to acquire an image of a message to read, extract the symbols and recognize them. Image character recognition research has been going on for decades now, with good results. But compared to conventional character recognition systems, the challenge with a mobile robot is to find a textual message to capture in the world and to get a good view of the message, knowing that the viewpoint of the robot depends on its position in relation to the message, which cannot be pre-specified. In this paper we present our approach making it possible for an autonomous mobile robot to read messages. We outline the constraints under which the approach works, and present results obtained using a Pioneer 2 robot equipped with a Pentium 233 MHz and a pan-tilt-zoom camera.

AZIMUT is a mobile robotic platform that combines wheels, legs and tracks to move in three-dimensional environments. The robot is symmetrical and is made of four independent leg-track-wheel articulations. It can move with its articulations up, down or straight, or to move sideways without changing the robot's orientation. To validate the concept, the first prototype developed measures 70.5 cm/spl times/70.5 cm with the articulations up. It has a body clearance of 8.4 cm to 40.6 cm depending on the position of the articulations. The design of the robot is highly modular, with distributed embedded systems to control the different components of the robot.

The hearing sense on a mobile robot is important because it is omnidirectional and it does not require direct line-of-sight with the sound source. Such capabilities can nicely complement vision to help localize a person or an interesting event in the environment. To do so the robot auditory system must be able to work in noisy, unknown and diverse environmental conditions. In this paper, we present a robust sound source localization method in three-dimensional space using an array of 8 microphones. The method is based on time delay of arrival estimation. Results show that a mobile robot can localize in real time different types of sound sources over a range of 3 meters and with a precision of 3/spl deg/.

With all the textual indications, messages and signs we find in urban settings to provide us with all types of information, it is only natural that we try to give to robots reading capabilities of interpreting such information. Equipped with optical character recognition algorithms, a mobile robot has to face the challenge of controlling its position in the world and its pan-tilt-zoom camera to find the textual message to capture, try to compensate for its viewpoint of the message, and use limited processing capabilities to decode the message. The robot also has to deal with non-uniform illumination and changing conditions in the world. In this work, we address the different aspects of the character recognition process to be incorporated into the higher level intelligence modules of a mobile robotic platform.

Recent research projects have demonstrated that it is possible to make robots move in formation. The approaches differ by the various assumptions about what can be perceived and communicated by the robots, the strategies used to make the robots move in formation, the ability to deal with obstacles and to switch formations. After suggesting criteria to characterize problems associated with robot formations, this paper presents a distributed approach based on directional visual perception and inter-robot communication. Using a pan camera head, sonar readings and wireless communication, we demonstrate that robots are not only able to move in formation, avoid obstacles and switch formations, but also initialize and determine by themselves their positions in the formation. Validation of our work is done in simulation and with Pioneer 2 robots.