Interaction understanding and activity recognition require the ability to identify patterns in sequences of events. A robot recording what is going on in real-life settings could be able to exploit these patterns using a representation similar to a map of the observed interactions. This paper examines the use of perceptual constructs, generated by deep neural networks that process visual and audio data, to build graph-based interaction mapping from the observation of human activities. Patterns are detected using loop closure detections, similarly to what is done in simultaneous localization and mapping approach. Results suggest that the graph-based interaction mapping approach is able to create condensed representations of interaction events, and find patterns in sequences of perceptual constructs that create loops. Such technique could help design socially intelligent robots that can derive an understanding of their environment from observing and learning from their interactions with people.

The performance of speech and events recognition systems significantly improved recently thanks to deep learning methods. However, some of these tasks remain challenging when algorithms are deployed on robots due to the unseen mechanical noise and electrical interference generated by their actuators while training the neural networks. Ego-noise reduction as a preprocessing step therefore can help solve this issue when using pre-trained speech and event recognition algorithms on robots. In this paper, we propose a new method to reduce ego-noise using only a microphone array and less than two minute of noise recordings. Using Principal Component Analysis (PCA), the best covariance matrix candidate is selected from a dictionary created online during calibration and used with the Minimum Variance Distortionless Response (MVDR) beamformer. Results show that the proposed method runs in real-time, improves the signal-to-distortion ratio (SDR) by up to 10 dB, decreases the word error rate (WER) by 55% in some cases and increases the Average Precision (AP) of event detection by up to 0.2.

A microphone array can provide a mobile robot with the capability of localizing, tracking and separating distant sound sources in 2D, i.e., estimating their relative elevation and azimuth. To combine acoustic data with visual information in real world settings, spatial correlation must be established. The approach explored in this paper consists of having two robots, each equipped with a microphone array, localizing themselves in a shared reference map using SLAM. Based on their locations, data from the microphone arrays are used to triangulate in 3D the location of a sound source in relation to the same map. This strategy results in a novel cooperative sound mapping approach using mobile microphone arrays. Trials are conducted using two mobile robots localizing a static or a moving sound source to examine in which conditions this is possible. Results suggest that errors under 0.3 m are observed when the relative angle between the two robots are above 30° for a static sound source, while errors under 0.3 m for angles between 40° and 140° are observed with a moving sound source.

For service robots to be well received in our daily lives, it is desirable that they appear as friendly as possible rather than some unfriendly characters. While a robot's physical appearance influences this perception, its behavior also has an impact. In order to be sure that a specific robot behavior will be correctly perceived, we propose to its potential users to shape the robot's behavior. In this paper, a specific behavior, “approaching a person”, is evaluated with a Robosoft Kompaï robot. To avoid logistics issues associated with having large groups of novice users performing demonstrations on a physical robot, a web-based approach built around a simulation of the actual robot is proposed. The relationship between the robot and the person is described by the two dimensions of the interpersonal circumplex: communion (hostile or friendly) and agency (submissive or dominant). The users can adjust three parameters of the approach behavior (i.e., distance, trajectory curvature, and deceleration) in a manner that corresponds the best to the described relationship. An analysis of the data from 69 users is presented, along with a verification experiment done with 10 participants and the real robot. Results suggest that users associate hostile robots with straight trajectories, and submissive robots with smoother deceleration.

A common goal of robot control architecture designers is to create systems that are sufficiently generic to be adapted to different robot hardware. Beyond code re-use from a software engineering standpoint, having a common architecture could lead to long-term experiments spanning multiple robots and research groups. This paper presents a first step toward this goal with HBBA, a Hybrid Behavior-Based Architecture first developed on the IRL-1 humanoid robot and integrating an Adaptive Resonance Theory-based episodic memory (EM-ART). This paper presents the first step of the adaptation of this architecture to two different robots, a Meka M-1 and a NAO from Aldebaran, with a simple scenario involving learning and sharing objects' information between both robots. The experiment shows that episodes recorded as sequences of people and objects presented to one robot can be recalled in the future on either robot, enabling event anticipation and sharing of past experiences.

Commercial telepresence robots provide video, audio, and proximity data to remote operators through a teleoperation user interface running on standard computing devices. As new modalities such as force sensing and sound localization are being developed and tested on advanced robotic platforms, ways to integrate such information on a teleoperation interface are required. This paper demonstrates the use of visual representations of forces and sound localization in a 3D teleoperation interface. Forces are represented using colors, size, bar graphs and arrows, while speech or ring bubbles are used to represents sound positions and types. Validation of these modalities is done with 31 participants using IRL-1/TR, a humanoid platform equipped with differential elastic actuators to provide compliance and force control of its arms and capable of sound source localization. Results suggest that visual representations of interaction force and sound source can provide appropriately useful information to remote operators.

Mobile devices such as smartphones and tablets can provide additional sensing and interacting capabilities to a mobile robot, even extending its senses to remote locations. To do so, we developed ROS4iOS, a native port of ROS allowing to seamlessly use data from mobile iOS devices to be processed on a robot. To demonstrate this capability, this video presentation illustrates how ROS4iOS has been used to implement an assistance scenario: a person in a remote location asks our IRL-1 robot for assistance, and IRL-1 must recognize the person's voice, identify the remote location using images taken from the mobile device, navigate to the identified location and interact vocally with the person through the mobile device. When communication with the robot is established by launching a specific iOS application, audio from the mobile device is published on a single topic and directed toward two ROS nodes on IRL-1. PocketSphinx, a speech recognition toolkit, is used to obtain the person vocal commands. WISS, a speaker identification system, is used for speaker identification. From a vocal request made through the mobile device, the robot can identify the person and request to get images of the remote location using the rear-facing camera of the mobile device. RTABMap, a loop-closure detection system for visual location recognition, is used to locate the person. A map was previously built with the robot's laser range finder using the gmapping SLAM algorithm, thus permitting reuse of the ROS navigation stack to plan a path and follow it safely from its current location to the requested one. Depending on its current state, the robot can also display the most suited view (i.e., phone, camera, navigation) on the iOS application. ROS4iOS can also be used to teleoperate IRL-1 through the mobile device or to discuss remotely with the robot, using the same ROS modules. ROS4iOS opens up a rich set of possibilities for HRI, making ROS-compatible code accessible on mobile devices. In future work, we plan to integrate a dialog management system for other assistance scenarios, such as image-based object fetching and delivery.

Smartphones and tablets are now part of our everyday lives, facilitating access to information through intuitive graphical interfaces. They can make great devices to work with robots because they integrate a lot of sensors and reasonable computing power. Using ROS on those devices would facilitate code reuse and integration with existing robotics applications and libraries.

Designing robots that interact naturally with people requires the integration of technologies and algorithms for communication modalities such as gestures, movement, facial expressions and user interfaces. To understand interdependence among these modalities, evaluating the integrated design in feasibility studies provides insights about key considerations regarding the robot and potential interaction scenarios, allowing the design to be iteratively refined before larger-scale experiments are planned and conducted. This paper presents three feasibility studies with IRL-1, a new humanoid robot integrating compliant actuators for motion and manipulation along with artificial audition, vision, and facial expressions. These studies explore distinctive capabilities of IRL-1, including the ability to be physically guided by perceiving forces through elastic actuators used for active steering of the omnidirectional platform; the integration of vision, motion and audition for an augmented telepresence interface; and the influence of delays in responding to sounds. In addition to demonstrating how these capabilities can be exploited in human-robot interaction, this paper illustrates intrinsic interrelations between design and evaluation of IRL-1, such as the influence of the contact point in physically guiding the platform, the synchronization between sensory and robot representations in the graphical display, and facial gestures for responsiveness when computationally expensive processes are used. It also outlines ideas regarding more advanced experiments that could be conducted with the platform.

The design and the evaluation of an autonomous interactive robot is a challenging research endeavor because there is as much to learn from the interaction between the integrated technologies as there is from the embodied human-robot interaction, in addition to observing their mutual interdependencies. This paper reports on IRL-0, a prototyping platform that we used to conduct preliminary studies on the influences of combining verbal and nonverbal modalities (facial expressions, head movement, arm gestures, and approach trajectory) for engaging interaction with people in controlled conditions and in real-world settings. IRL-0 is made of a compliant omnidirectional mobile base equipped with an expressive face and a three degrees-of-freedom (DOFs) compliant arm. By assembling this prototype and conducting these preliminary studies, our objective is to acquire insights in terms of design (e.g., technology, control) and experimental procedures that are important to take into consideration for the designing and evaluating autonomous robots engaging interaction with people.

Johnny-0, shown in Figure 1, is our new humanoid robot which integrates an expressive face on an orientable head, two arms with 4 degrees of freedom (DOF) each and grippers, mounted on an omnidirectional, non-holonomic mobile platform. Our underlying goal with Johnny-0 is to design a platform capable of natural reciprocal interaction (motion, language, touch, affect) with humans, to address integration issues associated with advanced motion, interaction and cognition capabilities on the same platform, and their use in unconstrained real world conditions. To do so, compliance is a necessity to provide natural and safe interactions.

AZIMUT-3 is an omnidirectional non-holonomic (or pseudo-omnidirectional) robotic platform intended for safe human-robot interaction. In its wheeled configuration, shown in Fig. 1, AZIMUT-3 uses eight actuators for locomotion: four for propulsion and four for steering the wheels, which can rotate 180 degrees around their steering axis. Propulsion is done using standard DC brushless motors (Bayside K064050-3Y) with optical encoders (US Digital E4-300-157-HUB, 0.3 deg of resolution), capable of reaching 1.47 m/s. The platform uses steerable wheels motorized using differential elastic actuators (DEA), which provide compliance, safety and torque control capabilities. AZIMUT-3's hardware architecture consists of distributed modules for sensing and low-level control, communicating with each other through a 1 Mbps CAN bus. A Mini-ITX computer equipped with a 2.0 GHz Core 2 duo processor running Linux with real-time patches (RT-PREEMPT) is used on-board for high-level control modules. Nickel-metal hydride batteries provide power to the platform for up to 3 hours of autonomy. A passive vertical suspension mechanism (Rosta springs) is used to connect the wheels to AZIMUT-3's chassis, allowing them to keep contact with the ground on uneven surfaces. The platform has a 34 kg payload capacity and weights 35 kg.

Robots are usually built using stiff actuators that can provide impressive motion performances. However, they struggle to control the force, they do not handle collisions graciously and are generally bad at interacting significantly with partially unknown or kinematically constrained environments. One solution is to add a force sensor in the closed-loop control of backdrivable actuators [1], but this is limited in terms of stability, safety and robustness [2] [3]. One of the foremost initiatives using this method was undertaken by the German Aerospace Center (DLR) and resulted in three generations of extensively optimized lightweight robotic arms [4] that can physically interact with people. Performances are impressive but robustness is still an issue.

The user interface is the central element of a telepresence robotic system and its visualization modalities greatly affect the operator's situation awareness, and thus its performance. Depending on the task at hand and the operator's preferences, going from ego- and exocentric viewpoints and improving the depth representation can provide better perspectives of the operation environment. Our system, which combines a 3D reconstruction of the environment using laser range finder readings with two video projection methods, allows the operator to easily switch from ego- to exocentric viewpoints. This paper presents the interface developed and demonstrates its capabilities by having 13 operators teleoperate a mobile robot in a navigation task.