Interaction understanding and activity recognition require the ability to identify patterns in sequences of events. A robot recording what is going on in real-life settings could be able to exploit these patterns using a representation similar to a map of the observed interactions. This paper examines the use of perceptual constructs, generated by deep neural networks that process visual and audio data, to build graph-based interaction mapping from the observation of human activities. Patterns are detected using loop closure detections, similarly to what is done in simultaneous localization and mapping approach. Results suggest that the graph-based interaction mapping approach is able to create condensed representations of interaction events, and find patterns in sequences of perceptual constructs that create loops. Such technique could help design socially intelligent robots that can derive an understanding of their environment from observing and learning from their interactions with people.

This paper introduces the Fast Cross-Correlation (FCC) method for Time Difference of Arrival (TDoA) Estimation for pairs of microphones on a small aperture microphone array. FCC relies on low-rank decomposition and exploits symmetry in even and odd bases to speed up computation while preserving TDoA accuracy. FCC reduces the number of flops by a factor of 4.5 and the execution speed by factors between 3.5 and 8.3 on embedded hardware, compared to the state-of-the-art Generalized Cross-Correlation (GCC) method that relies on the Fast Fourier Transform (FFT). This improvement can provide portable microphone arrays with extended battery life and allow real-time processing on low-cost hardware.

In recent years, studies on Socially Assistive Robots (SARs) examine how to improve the quality of life of people living with dementia and older adults (OAs) in general. However, most SARs have somewhat limited perception capabilities or interact using simple pre-programmed responses, providing limited or repetitive interaction modalities. Integrating more advanced perceptual capabilities with deep learning processing would help move beyond such limitations. This paper presents T-Top, a tabletop robot designed with advanced audio and vision processing using deep learning neural networks. T-Top is made available as an open source platform with the goal of providing an experimental SAR platform that can implement richer interaction modalities with OAs.

Socially Assistive Robots are foreseen as having the potential to improve the quality of life of older adults and individuals with mental disabilities. Natural human-robot interaction in everyday settings may require robots that are capable of understanding what is happening in their operating environments so that they can respond appropriately to the experienced situations and engage people in meaningful ways. This paper presents an approach using perceptual constructs to represent what is being observed by the robot. Perceptual constructs are derived from deep neural networks used to process visual and audio data. The objective is to derive a compressed representation of the interactions observed by the robot in real-life settings. Results are provided from observations made by a robot of a room with human activity over a two-week period, outlining what works and remaining challenges.

During these past years, Socially Assistive Robots (SARs) have been used to study the benefits of their uses with elderly people and people with dementia for healthcare purposes. Yet, almost all SARs have somewhat limited perception capabilities or respond using simple pre-programmed behaviors and reactions, providing limited or repetitive interaction modalities. To overcome these limitations and take into consideration the strengths and weaknesses of SARs in healthcare settings, this paper presents T-Top, a tabletop robot designed with advanced audio and vision sensors, deep learning perceptual processing and telecommunication capabilities. Designed as a open hard-ware/software platform, the objective with T-Top is to provide an experimental platform that can implement richer interaction modalities and develop higher cognitive abilities from interacting with people.

Distant speech processing is a challenging task, especially when dealing with the cocktail party effect. Sound source separation is thus often required as a preprocessing step prior to speech recognition to improve the signal to distortion ratio (SDR). Recently, a combination of beamforming and speech separation networks have been proposed to improve the target source quality in the direction of arrival of interest. However, with this type of approach, the neural network needs to be trained in advance for a specific microphone array geometry, which limits versatility when adding/removing microphones, or changing the shape of the array. The solution presented in this paper is to train a neural network on pairs of microphones with different spacing and acoustic environmental conditions, and then use this network to estimate a time-frequency mask from all the pairs of microphones forming the array with an arbitrary shape. Using this mask, the target and noise covariance matrices can be estimated, and then used to perform generalized eigenvalue (GEV) beamforming. Results show that the proposed approach improves the SDR from 4.78 dB to 7.69 dB on average, for various microphone array geometries that correspond to commercially available hardware.

A microphone array can provide a mobile robot with the capability of localizing, tracking and separating distant sound sources in 2D, i.e., estimating their relative elevation and azimuth. To combine acoustic data with visual information in real world settings, spatial correlation must be established. The approach explored in this paper consists of having two robots, each equipped with a microphone array, localizing themselves in a shared reference map using SLAM. Based on their locations, data from the microphone arrays are used to triangulate in 3D the location of a sound source in relation to the same map. This strategy results in a novel cooperative sound mapping approach using mobile microphone arrays. Trials are conducted using two mobile robots localizing a static or a moving sound source to examine in which conditions this is possible. Results suggest that errors under 0.3 m are observed when the relative angle between the two robots are above 30° for a static sound source, while errors under 0.3 m for angles between 40° and 140° are observed with a moving sound source.

In dynamic environments, performance of visual SLAM techniques can be impaired by visual features taken from moving objects. One solution is to identify those objects so that their visual features can be removed for localization and mapping. This paper presents a simple and fast pipeline that uses deep neural networks, extended Kalman filters and visual SLAM to improve both localization and mapping in dynamic environments (around 14 fps on a GTX 1080). Results on the dynamic sequences from the TUM dataset using RTAB-Map as visual SLAM suggest that the approach achieves similar localization performance compared to other state-of-the-art methods, while also providing the position of the tracked dynamic objects, a 3D map free of those dynamic objects, better loop closure detection with the whole pipeline able to run on a robot moving at moderate speed.

Rotary-Wing Air Vehicles (RW-UAVs), also referred to as drones, have gained in popularity over the last few years. Intrusions over secured areas have become common and authorities are actively looking for solutions to detect and localize undesired drones. The sound generated by the propellers of the RW-UAVs is powerful enough to be perceived by a human observer nearby. In this paper, we examine the use of particle filtering to detect and localize in 3D the position of a RW-UAV based on sound source localization (SSL) over distributed microphone arrays (MAs). Results show that the proposed method is able to detect and track a drone with precision, as long as the noise emitted by the RW-UAVs dominates the background noise.

To be used on a mobile robot, speech/non-speech discrimination must be robust to environmental noise and to the position of the interlocutor, without necessarily having to satisfy low-latency requirements. To address these conditions, this paper presents a speech/non-speech discrimination approach based on pitch estimation. Pitch features are robust to noise and reverberation, and can be estimated over a few seconds. Results suggest that our approach is more robust compared to the use of Mel-Frequency Cepstrum Coefficients with Gaussian Mixture Models (MFCC-GMM) under high reverberation levels and additive noise (with an accuracy above 98% with a latency of 2.21 sec), which makes it ideal for mobile robot applications. The approach is also validated on a mobile robot equipped with a 8-microphone array, using speech/non-speech discrimination based on pitch estimation as a post-processing module of a localization, tracking and separation system.

Sound source localization is an important challenge for mobile robots operating in real life settings. Sound sources of interest, such as speech, are often corrupted by broadband coherent noise sound source(s) that are non-stationary during transitions between steady-state segments. The interfering noise introduces localization ambiguities leading to the localization of invalid sound sources. Masks to reduce such interferences perform well under stationary noise, but the performance degrades as localization of invalid sound sources generated by noise appear and disappear suddenly during transitions between steady-state. This paper presents a new mask based on speech non-stationarity to discriminate between the time difference of arrival (TDOA) of speech source and noise transition. Simulations and experiments on a mobile robot suggest that the proposed technique improve TDOA discrimination and reduces significantly localization of invalid sound sources caused by noise.

Audition is a rich source of spatial, identity, linguistic and paralinguistic information. Processing all this information requires acquisition, processing and interpretation of sound sources, which are instantaneous, invisible and noisy signals. This can lead to different responses by the system in relation to the information perceived. This paper presents our first implementation of an integration framework for speech processing. Acquisition includes sound capture, sound source localization, tracking, separation and enhancement, and voice activity detection. Processing involves speech and emotion recognition. Interpretation consists of translating speech utterances into commands that can influence interaction through dialogue management and speech synthesis. The paper also describes two visualization interfaces, inspired by comic strips, to represent live vocal interactions in real life environments. These interfaces are used to demonstrate how the framework performs in live interactions and its use in a usability study.

Localization of sound sources in adverse environments is an important challenge in robot audition. The target sound source is often corrupted by coherent broadband noise, which introduces localization ambiguities as noise is often mistaken as the target source. To discriminate the time difference of arrival (TDOA) parameters of the target source and noise, this paper presents a binary mask for weighted generalized cross-correlation with phase transform (GCC-PHAT). Simulation and experiments on a mobile robot suggest that the proposed technique improves TDOA discrimination. It also brings the additional benefit of modulating the computing load requirement according to voice activity.

Tendon-Driven Manipulators (TDM) have been used in various applications requiring low inertia robots having multiple degrees-of-freedom (DOF) and redundancy. TDMs are generally actuated by gear electric motors placed at the base of the robot, and consequently require complex cable tension feedback from force sensors or dynamic modelling to maintain cables under tension. This paper presents a high performance TDM actuated by magneto-rheological clutches along with a specialized motion control algorithm, termed Projected PID, that requires no tension feedback. A 2-DOF proof-of-concept TDM powered by magneto-rheological clutches is used to demonstrate overall controller performance, and a reconfigurable 2-DOF TDM powered by direct-drive electric motors is used to demonstrate the controller's ability to compensate for configuration variations and actuator failure. Experimental results also demonstrate the ability of Projected PID to control magneto-rheological cable-driven TDMs with high accuracy.

Commercial telepresence robots provide video, audio, and proximity data to remote operators through a teleoperation user interface running on standard computing devices. As new modalities such as force sensing and sound localization are being developed and tested on advanced robotic platforms, ways to integrate such information on a teleoperation interface are required. This paper demonstrates the use of visual representations of forces and sound localization in a 3D teleoperation interface. Forces are represented using colors, size, bar graphs and arrows, while speech or ring bubbles are used to represents sound positions and types. Validation of these modalities is done with 31 participants using IRL-1/TR, a humanoid platform equipped with differential elastic actuators to provide compliance and force control of its arms and capable of sound source localization. Results suggest that visual representations of interaction force and sound source can provide appropriately useful information to remote operators.

Mobile devices such as smartphones and tablets can provide additional sensing and interacting capabilities to a mobile robot, even extending its senses to remote locations. To do so, we developed ROS4iOS, a native port of ROS allowing to seamlessly use data from mobile iOS devices to be processed on a robot. To demonstrate this capability, this video presentation illustrates how ROS4iOS has been used to implement an assistance scenario: a person in a remote location asks our IRL-1 robot for assistance, and IRL-1 must recognize the person's voice, identify the remote location using images taken from the mobile device, navigate to the identified location and interact vocally with the person through the mobile device. When communication with the robot is established by launching a specific iOS application, audio from the mobile device is published on a single topic and directed toward two ROS nodes on IRL-1. PocketSphinx, a speech recognition toolkit, is used to obtain the person vocal commands. WISS, a speaker identification system, is used for speaker identification. From a vocal request made through the mobile device, the robot can identify the person and request to get images of the remote location using the rear-facing camera of the mobile device. RTABMap, a loop-closure detection system for visual location recognition, is used to locate the person. A map was previously built with the robot's laser range finder using the gmapping SLAM algorithm, thus permitting reuse of the ROS navigation stack to plan a path and follow it safely from its current location to the requested one. Depending on its current state, the robot can also display the most suited view (i.e., phone, camera, navigation) on the iOS application. ROS4iOS can also be used to teleoperate IRL-1 through the mobile device or to discuss remotely with the robot, using the same ROS modules. ROS4iOS opens up a rich set of possibilities for HRI, making ROS-compatible code accessible on mobile devices. In future work, we plan to integrate a dialog management system for other assistance scenarios, such as image-based object fetching and delivery.

Smartphones and tablets are now part of our everyday lives, facilitating access to information through intuitive graphical interfaces. They can make great devices to work with robots because they integrate a lot of sensors and reasonable computing power. Using ROS on those devices would facilitate code reuse and integration with existing robotics applications and libraries.

For large-scale and long-term simultaneous localization and mapping (SLAM), a robot has to deal with unknown initial positioning caused by either the kidnapped robot problem or multi-session mapping. This paper addresses these problems by tying the SLAM system with a global loop closure detection approach, which intrinsically handles these situations. However, online processing for global loop closure detection approaches is generally influenced by the size of the environment. The proposed graph-based SLAM system uses a memory management approach that only consider portions of the map to satisfy online processing requirements. The approach is tested and demonstrated using five indoor mapping sessions of a building using a robot equipped with a laser rangefinder and a Kinect.

Recognizing a person from a distance is important to establish meaningful social interaction and to provide additional cues regarding the situations experienced by a robot. To do so, face recognition and speaker identification are biometrics commonly used, with identification performance that are influenced by the distance between the person and the robot. This paper presents a system that combines these biometrics with human metrology (HM) to increase identification performance and range. HM measures are derived from 2D silhouettes extracted online using a dynamic background subtraction approach, processing in parallel 45 front features and 24 side features in 400 ms compared to 38 front and 22 side features extracted in sequence in 30 sec by using the approach presented by Lin and Wang [1]. By having each modality identify a set of up to five possible candidates, results suggest that combining modalities provide better performance compared to what each individual modality provides, from a wider range of distances.

It is known that children with autism can benefit from interacting with robotic devices. The Centers for Disease Control and Prevention (CDC) in the USA identifies around 1 in 68 American children as being on the autism spectrum. Other countries also have similar prevalence rates. Therefore, providing therapeutic devices is becoming of increasing importance. Here we address the problem of effectively designing and building a robotic device fit for this purpose. The robot we have designed uses a simple spherical morphology together with a collection of autonomous behaviors and controllable modalities. The platform, is robust, simple, safe, and provides a wide range of movement, visual, sound and touch interaction capabilities to encourage the child to learn and play.

Designing robots that interact naturally with people requires the integration of technologies and algorithms for communication modalities such as gestures, movement, facial expressions and user interfaces. To understand interdependence among these modalities, evaluating the integrated design in feasibility studies provides insights about key considerations regarding the robot and potential interaction scenarios, allowing the design to be iteratively refined before larger-scale experiments are planned and conducted. This paper presents three feasibility studies with IRL-1, a new humanoid robot integrating compliant actuators for motion and manipulation along with artificial audition, vision, and facial expressions. These studies explore distinctive capabilities of IRL-1, including the ability to be physically guided by perceiving forces through elastic actuators used for active steering of the omnidirectional platform; the integration of vision, motion and audition for an augmented telepresence interface; and the influence of delays in responding to sounds. In addition to demonstrating how these capabilities can be exploited in human-robot interaction, this paper illustrates intrinsic interrelations between design and evaluation of IRL-1, such as the influence of the contact point in physically guiding the platform, the synchronization between sensory and robot representations in the graphical display, and facial gestures for responsiveness when computationally expensive processes are used. It also outlines ideas regarding more advanced experiments that could be conducted with the platform.

The design and the evaluation of an autonomous interactive robot is a challenging research endeavor because there is as much to learn from the interaction between the integrated technologies as there is from the embodied human-robot interaction, in addition to observing their mutual interdependencies. This paper reports on IRL-0, a prototyping platform that we used to conduct preliminary studies on the influences of combining verbal and nonverbal modalities (facial expressions, head movement, arm gestures, and approach trajectory) for engaging interaction with people in controlled conditions and in real-world settings. IRL-0 is made of a compliant omnidirectional mobile base equipped with an expressive face and a three degrees-of-freedom (DOFs) compliant arm. By assembling this prototype and conducting these preliminary studies, our objective is to acquire insights in terms of design (e.g., technology, control) and experimental procedures that are important to take into consideration for the designing and evaluating autonomous robots engaging interaction with people.

Johnny-0, shown in Figure 1, is our new humanoid robot which integrates an expressive face on an orientable head, two arms with 4 degrees of freedom (DOF) each and grippers, mounted on an omnidirectional, non-holonomic mobile platform. Our underlying goal with Johnny-0 is to design a platform capable of natural reciprocal interaction (motion, language, touch, affect) with humans, to address integration issues associated with advanced motion, interaction and cognition capabilities on the same platform, and their use in unconstrained real world conditions. To do so, compliance is a necessity to provide natural and safe interactions.

This paper presents WISS, a speaker identification system for mobile robots integrated to ManyEars, a sound source localization, tracking and separation system. Speaker identification consists in recognizing an individual among a group of known speakers. For mobile robots, performing speaker identification in presence of noise that changes over time is one important challenge. To deal with this issue, WISS uses Parallel Model Combination (PMC) and masks to update in real-time the speaker models (obtained in clean conditions) to both additive and convolutive noises. The results show that the weighted rate of good speaker identifications is 96% on average for a Signal-to-Noise Ratio (SNR) of 16 dB, whereas it only decreases to 84% when the SNR drops to 2 dB.

To demonstrate the influence of an artificial audition system on speech recognition and dialogue management for a robot, this paper presents a case study involving soft coupling of ManyEars, a sound source localization, tracking and separation system, with the CSLU Dialogue Management system. Trials were conducted in a laboratory and a cafeteria. Results indicate that preprocessing of the audio signals by ManyEars improves speech recognition and dialogue management of the system, demonstrating the feasibility and the added flexibility provided by ManyEars for a robot to interact vocally with humans in a wide variety of contexts.

Loop closure detection is the process involved when trying to find a match between the current and a previously visited locations in SLAM. Over time, the amount of time required to process new observations increases with the size of the internal map, which may influence real-time processing. In this paper, we present a novel real-time loop closure detection approach for large-scale and long-term SLAM. Our approach is based on a memory management method that keeps computation time for each new observation under a fixed limit. Results demonstrate the approach's adaptability and scalability using four standard data sets.

Redundantly actuated mobile robots using conventional wheels need a precise coordination of their actuators in order to guarantee a safe and precise motion without generating high internal forces and slippage. Using the instantaneous centre of rotation (ICR) of the chassis to describe this motion is a well established method. But the ICR is a mathematical concept which is hardly achieved on a real robot. This paper addresses the problem of ICR estimation of a non-holonomic omnidirectional mobile robot using conventional wheels. Instead of estimating the ICR in the working space, our approach estimates it in the actuators' space. The algorithm is presented in its general form and then adapted for a particular robot. The use of the algorithm with other omnidirectional robots is also discussed. Results from extensive testing done in simulation as well as with a real robot are presented, demonstrating the effectiveness of the proposed method.

Humans use direct physical interactions to move objects and guide people, and the same should be done with robots. However, most of today's mobile robots use non-backdrivable motors for locomotion, making them potentially dangerous in case of collision. This paper presents a robot, named AZIMUT-3, equipped with differential elastic actuators that are backdrivable and torque controlled, capable of being force-guided. Real world results demonstrate that AZIMUT-3 can move efficiently in response to physical commands given by a human pushing the robot in the intended direction.

Omnidirectional mobile robots, i.e., robots that can move in any direction without changing their orientation, offer better manoeuvrability in natural environments. Modeling the kinematics of such robots is a challenging problem and different approaches have been investigated. One of the best approaches for a nonholonomic robot is to model the robot's velocity state as the motion around its instantaneous center of rotation (ICR). In this paper, we present a motion planner designed to compute efficient trajectories for such a robot in an environment with obstacles. The action space is modeled in terms of changes of the ICR and the motion around it. Our motion planner is based on a Rapidly-Exploring Random Trees (RRT) algorithm to sample the action space and find a feasible trajectory from an initial configuration to a goal configuration. To generate fluid paths, we introduce an adaptive sampling technique taking into account constraints related to the ICR-based action space.

AZIMUT-3 is an omnidirectional non-holonomic (or pseudo-omnidirectional) robotic platform intended for safe human-robot interaction. In its wheeled configuration, shown in Fig. 1, AZIMUT-3 uses eight actuators for locomotion: four for propulsion and four for steering the wheels, which can rotate 180 degrees around their steering axis. Propulsion is done using standard DC brushless motors (Bayside K064050-3Y) with optical encoders (US Digital E4-300-157-HUB, 0.3 deg of resolution), capable of reaching 1.47 m/s. The platform uses steerable wheels motorized using differential elastic actuators (DEA), which provide compliance, safety and torque control capabilities. AZIMUT-3's hardware architecture consists of distributed modules for sensing and low-level control, communicating with each other through a 1 Mbps CAN bus. A Mini-ITX computer equipped with a 2.0 GHz Core 2 duo processor running Linux with real-time patches (RT-PREEMPT) is used on-board for high-level control modules. Nickel-metal hydride batteries provide power to the platform for up to 3 hours of autonomy. A passive vertical suspension mechanism (Rosta springs) is used to connect the wheels to AZIMUT-3's chassis, allowing them to keep contact with the ground on uneven surfaces. The platform has a 34 kg payload capacity and weights 35 kg.

Robots are usually built using stiff actuators that can provide impressive motion performances. However, they struggle to control the force, they do not handle collisions graciously and are generally bad at interacting significantly with partially unknown or kinematically constrained environments. One solution is to add a force sensor in the closed-loop control of backdrivable actuators [1], but this is limited in terms of stability, safety and robustness [2] [3]. One of the foremost initiatives using this method was undertaken by the German Aerospace Center (DLR) and resulted in three generations of extensively optimized lightweight robotic arms [4] that can physically interact with people. Performances are impressive but robustness is still an issue.

Most modern robotic systems are fast and repeatable position controlled machines. However, because of their inability to interact safely, robustly and in a versatile manner, they mostly remain confined to controlled areas where they execute specific preprogrammed actions. Providing high performance motion simultaneously with the ability to physically interact significantly remains a challenge.

Today's robotic systems are mostly rigid and position-controlled machines designed to operate in structured environments. To extend their application domains to partially unknown, dynamic, or anthropic environments, improved physical-interaction capabilities are required. In this new context, to blend the requirements for safety, robustness, and versatility is often a challenge, in part, because commonly available actuator technologies are inadequate. This paper presents our solution with the introduction of the dual-differential rheological actuator (DDRA) concept, which is based on the synergistic combination of an electromagnetic (EM) motor and two differentially coupled magnetorheological (MR) brakes. This paper describes the approach and the prototype design. It then discusses performances in force, motion, and interaction control.

Being able to act remotely in our homes could be very useful in providing various services such as surveillance and remote interventions, which are key features for telehomecare applications. In addition to navigation and environmental challenges that a telepresence robot would face in home settings, the system requires an appropriate teleoperation interface for safe and efficient usage by novice users. This paper describes the design criteria and characterizes visualization and control modalities of user interfaces with a real robot. By considering the user's needs along with the current state of the art in teleoperation interfaces, two novel mixed-reality visualization modalities are compared with standard video-centric and map-centric perspectives. We report teleoperation trials under six different task scenarios with a sample of 37 novice operators in homelike conditions. The results based on three quantitative metrics and one qualitative metric outline under which conditions the novel mixed-reality visualization modalities significantly improve the performance of novice users.

Providing healthcare in remote locations can prove to be costly. Using a static videoconference system in the patient's home has its limitations. A remotely operated mobile robot platform could provide a better interaction with the patient located at home. This paper presents Telerobot, a teleoperated mobile robotic platform equipped with videoconferencing capabilities. Developed by a team of roboticists and clinical experts, the system is designed specifically for the provision of in-home telerehabilitation services. A usability study was done in order to qualify the robot user control scheme and the clinician-patient interaction.

The user interface is the central element of a telepresence robotic system and its visualization modalities greatly affect the operator's situation awareness, and thus its performance. Depending on the task at hand and the operator's preferences, going from ego- and exocentric viewpoints and improving the depth representation can provide better perspectives of the operation environment. Our system, which combines a 3D reconstruction of the environment using laser range finder readings with two video projection methods, allows the operator to easily switch from ego- to exocentric viewpoints. This paper presents the interface developed and demonstrates its capabilities by having 13 operators teleoperate a mobile robot in a navigation task.

This video shows a light hearted view of a rolling autonomous robot named Roball. Roball is shown interacting with various children who age from 10 months old to teenagers at a high school. The clips show the different ways children interact with Roball and also the different types of reactions the children can have to Roball. Each clip was taken from a trial that was conducted to investigate Child-Robot Interaction (CRI).

There is great potential for robotic devices when being applied with children. They can be used from play to assistive applications. We develop robotic devices for a diverse range of children that differ in age, gender and ability, which includes children that are diagnosed with cognitive difficulties such as autism. Every child is an individual and they vary in their personalities and styles of interaction. Therefore, being able to adjust the robot's behaviour to the type of interaction it is receiving was believed to be essential. In this abstract we examine a series of trials which investigated how adaptation (through changes in motion and sound) on a fully autonomous rolling robot could help gain and sustain the interest of five different children. We discovered surprising benefits to having adaptation on-board Roball.

Although research on localization of sound sources using microphone arrays has been carried out for years, providing such capabilities on robots is rather new. Artificial audition systems on robots currently exist, but no evaluation of the methods used to localize sound sources has yet been conducted. This paper presents an evaluation of various real-time audio localization algorithms using a medium-sized microphone array which is suitable for applications in robotics. The techniques studied here are implementations and enhancements of steered response power - phase transform beamformers, which represent the most popular methods for time difference of arrival audio localization. In addition, two different grid topologies for implementing source direction search are also compared. Results show that a direction refinement procedure can be used to improve localization accuracy and that more efficient and accurate direction searches can be performed using a uniform triangular element grid rather than the typical rectangular element grid.

Auditory capabilities would allow small robots interacting with people to act according to vocal cues. In our recent work, we have demonstrated AUDIBLE, an auditory system capable of sound source localization, tracking and separation in real-time, using an array of eight microphones and running on a laptop computer. The system is able to localize and track up to four sources, while separating up to three sources in real-time in noisy environments. Signal processing techniques can be quite computer intensive, and the question of making it possible for this system to run on platforms that cannot carry a laptop computer onboard can be raised. This paper reports our investigation of the appropriate compromises to be made to AUDIBLE's implementation in order to port the system on an embedded DSP (Digital Signal Processor) platform. The DSP implementation is fully functional and performs well with minor limitations compared to the original system i.e., limitations on sound source duration and on the number of sources that can be processed simultaneously. Results demonstrate that it is feasible to port AUDIBLE on embedded platforms, opening up its use in field applications such as human-robot interaction in real life settings.

For complex robotic tasks (e.g., manipulation, locomotion), the lack of knowledge of precise interaction models, the difficulties to precisely measure the task associated physical quantities (e.g., position of contact points, interaction forces) in real-time, the finite sampling time of digital control loops and the non-collocation of sensors and transducers have negative effects on performance and stability of robots when using simple force or simple movement controllers. To cope with these issues, a new compact design for high performance actuators specifically adapted for integration in robotic mechanisms is presented. This design makes use of a mechanical differential as its central element. Results shown that differential coupling between an intrinsically high impedance transducer and an intrinsically low impedance mechanical spring provides the same benefits as serial coupling, but in a more compact and simple design. This new actuator, named Differential Elastic Actuator (DEA), provides interesting design implementations, especially for rotational actuators used for mobile robot locomotion.

Coordination of a group of mobile robots is facilitated when they are able to determine their positions relative to each other. Instead of using an absolute positioning approach with fixed beacons in the operating environment, we have developed a ultrasonic relative positioning system that allows each robot to perceive the distance and the angle of other nearby robots in relation to its own position. The system is based on time-of-flight evaluation of ultrasonic pulses and a RF communication link. The system has a precision of 8 mm and of 3deg over a 6.7 m range. This paper describes the system, its performance and its use on four Pioneer 2 robots moving in formation.

Behavior-based architectures use behaviors as building blocks for decision-making and action execution processes. Behaviors are distributed and evaluated in parallel for the control of the robot, taking real-time inputs from sensory data and sending real-time commands to effectors. No centralized components exist in these architectures, each module carrying out its own strategy independently, making an overall behavior emerge from the interaction between the concurrently executed modules and the environment. In this paper, we discuss the use of a reactive hierarchical task network (HTN) planner in a behavior-based robot architecture. The planner in this architecture is not a central component on which everything else relies on, but acts as one of the motivational modules recommending tasks to be executed and influencing the selection and configuration of behaviors. The planning module allows the behavior-based architecture to deal with tasks with priorities, flexible time constraints and on-line planning using a simple but very effective reactive planning strategy. We demonstrate our approach in the context of making a robot attend a conference.

The most common ground locomotion method to make a mobile robot move is to use two-wheel drive with differential steering and a rear balancing caster. Controlling the two motors independently makes the robot non-holonomic in its motion. Such robots can work well indoor on flat surfaces and in environments adapted for wheelchairs. But the benefit of providing mobility to a robot directly relies on its locomotion capability, for handling different types of terrains (indoors or outdoors) and situations such as moving slowly or rapidly, with or without the presence of moving objects (living or not), climbing over objects and potentially having to deal with hazardous conditions. It is with this objective in mind that we designed AZIMUT. AZIMUT is a legged tracked wheeled robot capable of changing the orientation of its four articulations. Each articulation has three degrees of freedom (DOF): it can rotate 360deg around its point of attachment to the chassis, can change its orientation over 180deg, and rotate to propulse the robot.

This video shows the functionalities of a 3 serial DOF robotic arm. Each DOF is actuated with a patent pending differential elastic actuator (DEA) [1,2]. Compared to the abundantly studied series elastic actuator [3,4], DEA uses a differential coupling between a high impedance mechanical speed source and a low impedance mechanical spring. Possible implementations of a mechanical differential include the use of a standard gearbox, harmonic drive, cycloidal gearbox, bar mechanism, cable mechanism and all other mechanism that implement a differential function between three mechanical ports. For the implementation reported in this video, we used a harmonic drive for a very compact design. A passive torsion spring (thus the name elastic), with a known impedance characteristic corresponding to the spring stiffness, is used, with an electrical DC brushless motor. A non-turning sensor connected in series with the spring measures the torque output of the actuator.

Increasingly researchers are looking outside of normal communication channels (such as video and audio) to provide additional forms of communication or interaction between a human and a robot, or a robot and its environment. Amongst the new channels being investigated is the detection of touch using infrared, proprioceptive and temperature sensors. Our work aims at developing a system that can detect natural touch or interaction coming from children playing with a robot, and adapt to this interaction. This paper reports trials carried out using Roball, a spherical mobile robot, demonstrating how sensory data patterns can be identified in human-robot interaction, and exploited for achieving behavioral adaptation. The experimental methodology used for these trials is reported, which validated the hypothesis that human interaction can not only be perceived from proprioceptive sensors on-board a robotic platform, but that this perception has the ability to lead to adaptation.

This paper describes a system that gives a mobile robot the ability to perform automatic speech recognition with simultaneous speakers. A microphone array is used along with a real-time implementation of geometric source separation (GSS) and a postfilter that gives a further reduction of interference from other sources. The postfllter is also used to estimate the reliability of spectral features and compute a missing feature mask. The mask is used in a missing feature theory-based speech recognition system to recognize the speech from simultaneous Japanese speakers in the context of a humanoid robot. Recognition rates are presented for three simultaneous speakers located at 2 m from the robot. The system was evaluated on a 200-word vocabulary at different azimuths between sources, ranging from 10deg to 90deg. Compared to the use of the microphone array source separation alone, we demonstrate an average reduction in relative recognition error rate of 24% with the postfllter and of 42% when the missing features approach is combined with the postfllter. We demonstrate the effectiveness of our multisource microphone array postfilter and the improvement it provides when used in conjunction with the missing features theory.

In this paper we present a new robust sound source localization and tracking method using an array of eight microphones (US patent pending). The method uses a steered beamformer based on the reliability-weighted phase transform (RWPHAT) along with a particle filter-based tracking algorithm. The proposed system is able to estimate both the direction and the distance of the sources. In a videoconferencing context, the direction was estimated with an accuracy better than one degree while the distance was accurate within 10% RMS. Tracking of up to three simultaneous moving speakers is demonstrated in a noisy environment

This paper describes the functionalities of ARMUS, an ARM robotic processing system designed by a team of fourth-year undergraduate students in electrical engineering and computer engineering. This project intends to replace the Handy Board, the current microcontroller system used by first-year undergraduate students in our curricula with a more powerful, versatile and up-to-date technology, while preserving ease of use. Our ARMUS processing system offers high processing and memory capabilities at low energy consumption, for a price of around 300$CAD

Mobile robots operating in home environments must deal with constrained space and a great variety of obstacles and situations to handle. This article presents a pilot study aiming at identifying design specifications of a new user interface and robot specifications to improve efficiency and security for novice teleoperators of a mobile robot used in home environments. This pilot study is part of the familiarization phase of an iterative interdisciplinary design process aiming at outlining critical design and experimental issues before engaging into detailed design processes, elaborated experimental methodology and rigorous testing of the various capabilities of mobile robots for home care applications. We evaluated, with a small set of trained and untrained operators, two conceptually different user interfaces for teleoperated mobile robotic systems. These results demonstrate the challenges and the necessity of conducting trials in home environments to evaluate such teleoperated systems, and outline distinct preferences regarding robot capabilities, user interface navigation method and evaluation methodology

To eventually have automated vehicles operate in platoons, it is necessary to study what information each vehicle must have and to whom it must communicate for safe and efficient maneuvering in all possible conditions. This paper formulates the problem in terms of sensing and communicated information. By emulating platoons using a group of mobile robots, the authors demonstrate the feasibility of maneuvers (such as entering, exiting, and recuperating from an accident) using different distributed coordination strategies. The coordination strategies studied range from no communication to unidirectional or bidirectional exchanges between vehicles and to fully centralized decision by the leading vehicle. One particularity of this paper is that instead of assuming that the platoon leader or all vehicles globally monitor what is going on, only the vehicles involved in a particular maneuver are concerned, distributing decisions locally among the platoon. This paper reports experimental trials using robots having limited and directional perception of other things, using vision and obstacle avoidance sensing. Results confirm the feasibility of the coordination strategies in different conditions and various uses of communicated information to compensate for sensing limitations

A humanoid robot under real-world environments usually hears mixtures of sounds, and thus three capabilities are essential for robot audition; sound source localization, separation, and recognition of separated sounds. While the first two are frequently addressed, the last one has not been studied so much. We present a system that gives a humanoid robot the ability to localize, separate and recognize simultaneous sound sources. A microphone array is used along with a real-time dedicated implementation of Geometric Source Separation (GSS) and a multi-channel post-filter that gives us a further reduction of interferences from other sources. An automatic speech recognizer (ASR) based on the Missing Feature Theory (MFT) recognizes separated sounds in real-time by generating missing feature masks automatically from the post-filtering step. The main advantage of this approach for humanoid robots resides in the fact that the ASR with a clean acoustic model can adapt the distortion of separated sound by consulting the post-filter feature masks. Recognition rates are presented for three simultaneous speakers located at 2m from the robot. Use of both the post-filter and the missing feature mask results in an average reduction in error rate of 42% (relative).

A humanoid robot under real-world environments usually hears mixtures of sounds, and thus three capabilities are essential for robot audition; sound source localization, separation, and recognition of separated sounds. We have adopted the missing feature theory (MFT) for automatic recognition of separated speech, and developed the robot audition system. A microphone array is used along with a real-time dedicated implementation of geometric source separation (GSS) and a multi-channel post-filter that gives us a further reduction of interferences from other sources. The automatic speech recognition based on MFT recognizes separated sounds by generating missing feature masks automatically from the post-filtering step. The main advantage of this approach for humanoid robots resides in the fact that the ASR with a clean acoustic model can adapt the distortion of separated sound by consulting the post-filter feature masks. In this paper, we used the improved Julius as an MFT-based automatic speech recognizer (ASR). The Julius is a real-time large vocabulary continuous speech recognition (LVCSR) system. We performed the experiment to evaluate our robot audition system. In this experiment, the system recognizes a sentence, not an isolated word. We showed the improvement in the system performance through three simultaneous speech recognition on the humanoid SIG2.

This paper presents the design process of a spherical robot capable of autonomous motion, and demonstrates how it can become a tool in child-development studies. The robot, named Roball, is capable of intentional self-propelled movements and can generate various interplay situations using motion, messages, sounds, illuminated parts and other sensors. Such capabilities allow Roball to interact with young children in simple and interesting ways, and to provide the potential of contributing to the development of their language, affective, motor, intellectual and social skills. Trials done with 12-24-month-old children demonstrate how Roball can be used to study children's interest in a self-propelled and intentional device. An experimental methodology to conduct such studies is presented: it is based on quantitative and qualitative techniques to evaluate interactions, thus enabling the identification of challenges and opportunities in child-robot interaction studies.

Microphone array post-filters have demonstrated their ability to greatly reduce noise at the output of a beamformer. However, current techniques only consider a single source of interest, most of the time assuming stationary background noise. We propose a microphone array post-filter that enhances the signals produced by the separation of simultaneous sources using common source separation algorithms. Our method is based on a loudness-domain optimal spectral estimator and on the assumption that the noise can be described as the sum of a stationary component and of a transient component that is due to leakage between the channels of the initial source separation algorithm. The system is evaluated in the context of mobile robotics and is shown to produce better results than current post-filtering techniques, greatly reducing interference while causing little distortion to the signal of interest, even at very low SNR.

Real life deployment of robot formation cannot assume that robots are going to be correctly positioned to move in a particular configuration. To do so, we propose an approach that allows the group to determine autonomously the most appropriate assignment of positions in the formation. Our approach is distributed and uses directional visual perception to localize robots. Inter-robot communication allows them to share information on which robots are nearby, so that each can evaluate it ability to be the conductor of the group and assign formation positions to the other robots by minimizing repositioning. The assignment search is done using a distributed bounded depth-first with pruning search. The robot with the best score is selected as the conductor, and the other robots receive from the conductor their assignment in the formation. Validation of our work is done in simulation and with Pioneer 2 robots.

Mobile robots in real-life settings would benefit from being able to localize sound sources. Such a capability can nicely complement vision to help localize a person or an interesting event in the environment, and also to provide enhanced processing for other capabilities such as speech recognition. We present a robust sound source localization method in three-dimensional space using an array of 8 microphones. The method is based on a frequency-domain implementation of a steered beamformer along with a probabilistic post-processor. Results show that a mobile robot can localize in real time multiple moving sources of different types over a range of 5 meters with a response time of 200 ms.

This paper describes two initiatives aiming at improving code reusability for programming mobile robots: robotflow/flowdesigner, a data-flow programming environment; MARIE (mobile and autonomous robotics integration environment), a programming environment allowing multiple applications, programs and tools, to operate on one or multiple machines/OS and work together on a mobile robot implementation. Robotflow/flowdesigner's objective is to provide a modular, graphical programming environment that would help visualize and understand what is really happening in the robot's control loops, sensors, actuators, by using graphical probes. MARIE aims at avoiding making an exclusive choice on particular programming tools, making it possible to reuse code and applications.

We propose a system that gives a mobile robot the ability to separate simultaneous sound sources. A microphone array is used along with a real-time dedicated implementation of geometric source separation and a post-filter that gives us a further reduction of interferences from other sources. We present results and comparisons for separation of multiple non-stationary speech sources combined with noise sources. The main advantage of our approach for mobile robots resides in the fact that both the frequency domain geometric source separation algorithm and the post-filter are able to adapt rapidly to new sources and non-stationarity. Separation results are presented for three simultaneous interfering speakers in the presence of noise. A reduction of log spectral distortion (LSD) and increase of signal-to-noise ratio (SNR) of approximately 10 dB and 14 dB are observed.

For robots to move in formation or to make them cooperate for distributed sensing of an area, they need to be able to discriminate their position relative to each other. Instead of using an absolute positioning approach, we have developed a relative positioning device that allows to perceive the distance and the angle of other nearby devices. The system is based on time-of-flight evaluation of ultrasonic waves and a RF communication link. The approach is validated using two Pioneer 2 robots in a leader-follower configuration.

Giving the ability to read characters and symbols is highly desirable for increased autonomy of mobile robots operating in the real world. The idea is fairly simple: give a robot the ability to acquire an image of a message to read, extract the symbols and recognize them. Image character recognition research has been going on for decades now, with good results. But compared to conventional character recognition systems, the challenge with a mobile robot is to find a textual message to capture in the world and to get a good view of the message, knowing that the viewpoint of the robot depends on its position in relation to the message, which cannot be pre-specified. In this paper we present our approach making it possible for an autonomous mobile robot to read messages. We outline the constraints under which the approach works, and present results obtained using a Pioneer 2 robot equipped with a Pentium 233 MHz and a pan-tilt-zoom camera.

AZIMUT is a mobile robotic platform that combines wheels, legs and tracks to move in three-dimensional environments. The robot is symmetrical and is made of four independent leg-track-wheel articulations. It can move with its articulations up, down or straight, or to move sideways without changing the robot's orientation. To validate the concept, the first prototype developed measures 70.5 cm/spl times/70.5 cm with the articulations up. It has a body clearance of 8.4 cm to 40.6 cm depending on the position of the articulations. The design of the robot is highly modular, with distributed embedded systems to control the different components of the robot.

The hearing sense on a mobile robot is important because it is omnidirectional and it does not require direct line-of-sight with the sound source. Such capabilities can nicely complement vision to help localize a person or an interesting event in the environment. To do so the robot auditory system must be able to work in noisy, unknown and diverse environmental conditions. In this paper, we present a robust sound source localization method in three-dimensional space using an array of 8 microphones. The method is based on time delay of arrival estimation. Results show that a mobile robot can localize in real time different types of sound sources over a range of 3 meters and with a precision of 3/spl deg/.

The long-term goal of this project is to derive systems that would allow the safe and efficient coordination of collaborating vehicles in high-density highway traffic in order to alleviate traffic congestion and reduce driving stress. The challenge is to ensure safe movements of each vehicle, inside the collaborative driving system. An architecture for the control and collaboration of vehicles is needed. We will present the importance of communication in CDS. After reviewing related work, the control scenarios for making vehicles join a pre-existing platoon, leave a platoon, join two platoons, do lane transition of a platoon or for ensuring safe emergency procedures are presented. The architecture developed to assure the safe execution of those scenarios is then exposed.

This paper describes the decentralized longitudinal control of a ten-vehicle platoon. An adapted time-headway distance approach is adopted, i.e., that each vehicle has to respect a "time-to-contact" with its preceding vehicle. The adaptation is made relatively to the vehicle dynamics and the road conditions. A two-level longitudinal controller is developed using the linear quadratic regulator (LQR) and feedback linearization methods. String stability analysis is discussed resulting in a definition for the adopted control strategy. A pentic polynomial guidance algorithm is used to limit the acceleration when a vehicle is reaching a platoon. Simulations showing the leader velocity variation for normal, icy and snowy roads and the pentic guidance approach are presented.

With all the textual indications, messages and signs we find in urban settings to provide us with all types of information, it is only natural that we try to give to robots reading capabilities of interpreting such information. Equipped with optical character recognition algorithms, a mobile robot has to face the challenge of controlling its position in the world and its pan-tilt-zoom camera to find the textual message to capture, try to compensate for its viewpoint of the message, and use limited processing capabilities to decode the message. The robot also has to deal with non-uniform illumination and changing conditions in the world. In this work, we address the different aspects of the character recognition process to be incorporated into the higher level intelligence modules of a mobile robotic platform.

Pervasive developmental disorders (PDD) refers to a group of disorders characterized by delays in the development of multiple basic functions including socialization and communication. Symptoms may include communication problems such as using and understanding language; difficulty relating to people, objects, and events; unusual play with toys and other objects; difficulty with changes in routine or familiar surroundings, and repetitive body movements or behavior patterns. Autism is the most characteristic and best studied PDD. We are investigating the use of mobile robotic toys that can move in the environment and interact in various manners (vocal messages, music, visual cues, movement, etc.) with children with autism. The hypothesis is that mobile robots can serve as an appropriate pedagogical tool to help children with PDD develop social skills because they are more predictable and less intimidating. The objective is to see how such devices can be used to capture the child's attention and contribute to helping him or her develop social skills. This paper outlines the design considerations for such robots, and presents experimental protocols that are being developed to study the impacts of using these robots on the development of the child.

Recent research projects have demonstrated that it is possible to make robots move in formation. The approaches differ by the various assumptions about what can be perceived and communicated by the robots, the strategies used to make the robots move in formation, the ability to deal with obstacles and to switch formations. After suggesting criteria to characterize problems associated with robot formations, this paper presents a distributed approach based on directional visual perception and inter-robot communication. Using a pan camera head, sonar readings and wireless communication, we demonstrate that robots are not only able to move in formation, avoid obstacles and switch formations, but also initialize and determine by themselves their positions in the formation. Validation of our work is done in simulation and with Pioneer 2 robots.

To operate over a long period of time, autonomous mobile robots must have the capability of recharging themselves whenever necessary. In addition to be able to find and connect to a power source, robots must also consider taking actions to preserve and share energy in an environment where energy is a limited resource. Coordination is then required to ensure the survival of the group and the accomplishment of the robots' tasks. This paper explores these issues by allowing robots to predict and reason about their energetic capabilities, as individuals and as a group. The approach described allows robots to determine when to recharge, when to change their activity level and how long they should recharge. Validation of the work is done in simulation to demonstrate the versatility of the approach for different numbers of robots and power sources. Experiments with Pioneer 2 robots are also reported.

Intelligence has been an object of study for a long time. Different architectures try to capture and reproduce these aspects into artificial systems (or agents), but there is still no agreement on how to integrate them into a general framework. With this objective in mind, we propose an architectural methodology based on the idea of intentional configuration of behaviors. Behavior‐producing modules are used as basic control components that are selected and modified dynamically according to the intentions of the agent. These intentions are influenced by the situation perceived, knowledge about the world, and internal variables that monitor the state of the agent. The architectural methodology preserves the emergence of functionality associated with the behavior‐based paradigm in the more abstract levels involved in configuring the behaviors. Validation of this architecture is done using a simulated world for mobile robots, in which the agent must deal with various goals such as managing its energy and its well‐being, finding targets, and acquiring knowledge about its environment. Fuzzy logic, a topologic map learning algorithm, and activation variables with a propagation mechanism are used to implement the architecture for this agent.

Presents an automated system for counting fish by species. This system is to be used in fishways for monitoring and surveying fish. The system requires very few adjustments and no special installation. An infrared silhouette sensor is used to acquire the fish silhouettes These silhouettes are then processed on a personal computer for fish counting and classification by species. The system allows the operator to select the species of interest according to the fauna of the specified river. Classification is made based on the combined results of a Bayes maximum likelihood classifier, a learning vector quantization classifier and a one-class-one-network neural network classifier. Through the use of specialized classifiers of different types, a robust, modular and expandable recognition system is created.

This paper presents an integrated sensor fusion strategy for the positioning of an underwater ROV. The use of multiple Kalman filters makes the system highly adaptable by allowing different combinations of sensors without any modification of the models. This algorithm can handle any number of redundant sensors by using multi-filter fusion and can work asynchronously with different sensor data rates through a filter switching process.

This paper presents EME, a lour-cost solution for a highly modular multiprocessing hardware-software environment that can be useful in designing autonomous mobile robots. High processing capabilities, the ability to interface a large number of sensors and actuators, extensibility, redundancy and flexibility are issues addressed by EME. An autonomous robot equipped with two cameras, one for line-following and the other for visual communication by light signals, is used to demonstrate EME capabilities and performances.

Learning in the mobile robot domain is a very challenging task, especially in nonstationary conditions. This paper presents an approach that allows a robot to learn a model of its interactions with its operating environment in order to manage them according to the experienced dynamics. The robot is initially given a set of "behavior-producing" modules to choose from, and the algorithm provides a means of making that choice intelligently and dynamically. The approach is validated using a vision- and sonar-based Pioneer I robot in non-stationary conditions, in the context of a multirobot foraging task. Results show the effectiveness of the approach in taking advantage of any regularities experienced in the world, leading to fast and adaptable specialization for the learning robot.

Video line doubling can be realized using time-space interpolation filters. To improve their performances for moving diagonal lines, we have developed a fuzzy edge-direction detector. This fuzzy detector works by identifying small pixel variations in five orientations (0/spl deg/, /spl plusmn/45/spl deg/, and /spl plusmn/60/spl deg/) and by using rules to infer the prevailing direction. This direction is then used to spatially rotate the interpolation filter. For the fuzzy detector, three fuzzy sets are used to characterize the inputs, and two rule bases have been validated. This article presents the characteristics of the fuzzy edge-direction detector along with the methodology used for fuzzy sets positioning. Detection and interpolation results are also presented.

Artificial neural networks (ANN) have been demonstrated to be increasingly more useful for complex problems difficult to solve with conventional methods. With their learning abilities, they avoid having to develop a mathematical model or acquiring the appropriate knowledge to solve a task. The difficulty now lies in the ANN design process. A lot of choices must be made to design an ANN, and there are no available design rules to make these choices directly for a particular problem. Therefore, the design for an ANN demands a certain number of iterations, mainly guided by the expertise and the intuition of the developer. To automate the ANN design process, we have developed Neurex, composed of an expert system and an ANN simulator. Neurex autonomously guides the iterative ANN design process. Its structure tries to reproduce the design steps done by a human expert in conceiving an ANN. As a whole, the Neurex structure serves as a framework to implement this expertise for different learning paradigms. This article presents the system's general characteristics and its use in designing ANN using the standard backpropagation learning law.