We propose a proactive teleoperation support system to help high-power individuals exercise power appropriately during hierarchical interactions. In our study, seniors (high-power individuals) remotely operated an avatar robot as operators, while juniors (low-power individuals) exercised as exercisers. To ground the design, we conducted an observational study of hierarchical interactions, identifying three recurring challenges: perceived loss of power, escalation of negativity, and abandonment of power. Based on these findings, we proposed interaction policies and integrated them into a teleoperation system that provides operators with guidance for the appropriate use of power, aiming to prevent conflict and foster positive communication. We evaluated this system in a 2-hour study with 17 senior–junior pairs, comparing it to a baseline teleoperation system where operators controlled the avatar robot directly without such guidance. Results showed that our system significantly improved operator satisfaction and reduced workload. Exercisers reported greater enjoyment and acceptance, with similar exercise counts in both conditions. Interviews revealed that the system broadened operators’ communication strategies and fostered more positive, supportive environments for low-power participants.

Telepresence technology creates the opportunity for people that were traditionally left out of the workforce to work remotely. In the service industry, a pool of novice remote workers could teleoperate robots to perform short work stints to fill in the gaps left by the dwindling workforce. A hurdle is that consistently talking appropriately and politely imposes a severe mental burden on such novice operators and the quality of the service may suffer. In this study, we propose a teleoperation support system that lets novice remote workers talk freely without considering appropriateness and politeness while maintaining the quality of the service. The proposed system exploits intent recognition to transform casual utterances into predefined appropriate and polite utterances. We conducted a within-subject user study where 23 participants played the role of novice remote operators controlling a guardsman robot in charge of monitoring customers’ behaviors. We measured the workload with and without using the proposed support system using NASA task load index questionnaires. The workload was significantly lower ( p < .001) when using the proposed support system ( M = 46.07, SD = 14.36) than when not using it ( M = 62.74, SD = 12.70). The effect size was large (Cohen’s d = 1.23).

In this paper, we propose a system that supports operators who provide services to customers using teleoperated robots. We observed that unprofessional or lazy operators of teleoperated robots are a risk for businesses as they are likely to speak in ways that are inappropriate for customer services. The proposed system lets competent operators talk freely to customers and thus provide high quality service. For subpar operators, the proposed system filters inappropriate utterances to improve the service they provide. We conducted a user study with 21 participants to compare the proposed support system to a baseline system where operators talk freely to customers. For subpar operators, the quality of the service is significantly higher in terms of perceived politeness and reported customer satisfaction when using the proposed support system compared to when using the baseline system. For competent operators, we found no significant differences in the quality of the service between the two systems.

The service industry is facing an increase in the number of malicious customers (customers with unreasonable complaints). Employees have reported that handling unreasonable complaints is particularly stressful. Considering the recent push for workplace automation, we should have robots handling this task in place of humans. We propose a robot behavioral model designed for handling unreasonable complaints. The robot with this model has to "please the customer'' without proposing a settlement. From a large survey of Japanese workers conducted by labor unions and an interview survey of experienced workers we conducted, we identified the conventional complaint handling flow as 1) listen to the complaint, 2) confirm the content of the complaint, 3) apologize, 4) give an explanation, and 5) conclude. The proposed behavioral model is a variation of this flow that takes into account the "state of mind'' of the customer. In particular, the robot with this model does not leave the first step and keeps asking questions until the customer is "ready to listen''. We conducted a user study, using a Wizard-of-Oz approach, to compare the proposed behavioral model to a baseline one implementing the conventional flow. We replicated in our laboratory the situation of a customer in a mobile phone shop. The proposed behavioral model was significantly better at making the customers believe that the robot listened to them and tried to help.

This paper presents a robot behavioral model designed to support children during self-study. In particular, we want to investigate how a robot could increase the time children keep concentration. The behavioral model was developed by observing children during self-study and by collecting information from experienced tutors through interviews. After observing the children, we decided to consider three states corresponding to different levels of concentration. The child can be smoothly performing the task (“learning” state), encountering some difficulties (“stuck” state) or distracted (“distracted” state). The behavioral model was designed to increase the time spent concentrating on the task by implementing adequate behaviors for each of these three states. These behaviors were designed using the advices collected during the interview survey of the experienced tutors. A self-study system based on the proposed behavior model was implemented. In this system, a small robot sits on the table and encourages the child during self-study. An operator is in charge of determining the state of the child (Wizard of Oz) and the behavioral model triggers the appropriate behaviors for the different states. To demonstrate the effectiveness of the proposed behavioral model, a user study was conducted: 22 children were asked to solve problems alone and to solve problems with the robot. The children spent significantly (p = 0.024) more time in the “learning” state when studying with the robot.

This paper presents a model of navigational habituation for human passengers riding an autonomous robotic wheelchair. We present an example on how robot technology is utilized to understand human behavior which is useful to build human-like navigation models. The work consists of three parts, data collection from manual driving participants, habituation modeling and autonomous navigation experimentation. Manual driving data collection consisted of recording human driving control behavior using a robotic wheelchair in a complex labyrinth-like environment, while measuring physiological activity and emotional state via subjective reports (questionnaires). The purpose of this first part was to collect a sufficient representation of the evolution of human driving behavior in a complex environment (23 participants), such that a numerical model could be found to approximate the trend in velocity usage; velocity usage characterizes navigational habituation. The complex environment habituation model was tested and evaluated, on a different set of 16 participants, in a fully autonomous navigation setting where the robotic wheelchair followed a pre-recorded path. Self-reports and physiological measures confirmed that the proposed habitation velocity regulation model provides a more comfortable ride compared to slow and fast pre-defined static velocity profiles.

This paper presents a two-step framework for creating the three-dimensional (3-D) sound map of an environment with a mobile robot. The first step is the creation of a map that describes the geometry of the environment. The second step is the addition of the acoustic information to the geometric map. The result is a sound map that shows the probability of emitting sound for all the structures in the environment. To build the sound map, a mobile robot equipped with a microphone array drives through the mapped environment. During this drive, the acoustic information gathered by the microphone array is accumulated in a probabilistic manner. First, the likelihood of sound source presence in a set of directions is evaluated from the acoustic power received from these directions. Then, using an estimate of the robot's pose, an acoustic ray casting procedure transfers this likelihood to the structures in the geometric map. Finally, the probability that these structures emit sound is updated accordingly to the likelihood. Experimental results show that the sound maps are: accurate as it was possible to localize sound sources in 3-D and practical as different types of environments were mapped.

In order to solve the problems of current hearing aid devices, we make use of environment sensor network, and propose a hearing support system, where individual target and anti-target sound sources in the environment can be selected, and spatial information of the target sound sources is reconstructed. The performance of the selective sound separation module was evaluated for different noise conditions. Results showed that signal-to-noise ratios of around 15dB could be achieved by the proposed system for a 65dB babble noise plus directional music noise condition. In the same noise condition, subjective intelligibility tests were conducted, and an improvement of 65 to 90% word intelligibility rates could be achieved by using the proposed hearing support system.

This work proposes a Human-Comfortable Path Planner (HCoPP) system for autonomous passenger vehicles. The aim is to create a path planner that improves the feeling of comfort of the passenger, this topic is different from collision free planning and it has not received much attention. For this purpose, in addition to the shortest distance constraint conventionally used in path planning, constraints related to relevant environmental features are introduced. For straight segments, the constraint is based on the lane-circulation pattern preferred by humans. In curved segments and intersections, the constraint takes into account the visibility. A multi-layered cost map is proposed to integrate these additional constraints. To compute the human-comfortable path, a graph search algorithm was implemented. The evaluation of the proposed approach was conducted by having 30 participants riding an autonomous robotic wheelchair. The paths computed by the proposed path planner were compared towards a state of the art shortest-distance path planner implemented in the navigation stack of ROS. Experimental results show that the paths computed by the proposed approach are perceived as more comfortable.

This paper presents a method for representing acoustic information with point clouds by tying it to geometrical features. The motivation is to create a representation of this information that is well suited for mobile robotic applications. In particular, the proposed approach is designed to take advantage of the use of multiple coordinate frames. As an illustrative example, we present a way to create an audio augmented point cloud by adding estimated audio power to the point cloud created by a RGB-D camera. A few applications of this method are presented.

We developed a system for detecting the speech activity intervals of multiple speakers by combining multiple microphone arrays and human tracking technologies. We also proposed a method for estimating the face orientation of the detected speakers. The developed system was evaluated in two steps: individual utterances in different positions and orientations; and simultaneous dialogues by multiple speakers. Evaluation results revealed that the proposed system could detect speech activity intervals with more than 90% of accuracy, and face orientations with standard deviations within 30 degrees, in situations excluding the cases where all arrays are in the opposite direction to the speaker's face orientation.

This paper presents a work on mapping the use of space by humans in long periods of time. Daily geometric maps with the same coordinate frame were generated with SLAM, and in a similar manner, daily affordance density maps (places people use) were generated with the output of a human tracker running on the robot. The contribution of the paper is two-fold: an approach to detect geometric changes to cluster them in similar geometric configurations and the building of geometric and affordance composite maps on each cluster. This approach avoids the loss of long term retrieved information. Geometric similarity was computed using a normal distance approach on the maps. The analysis was performed on data collected by a mobile robot for a period of 4 months accumulating data equivalent to 70 days. Experimental results show that the system is capable of detecting geometric changes in the environment and clustering similar geometric configurations.

This work presents a human-robot cooperative approach for infrastructure inspection. The goal is to create a robot that assists the human inspector during hammer sounding inspections. Hammer sounding is a frequently used inspection technique that detects invisible defects under the surface of concrete by striking the surface with a hammer and listening the resulting sound. The conventional hammer sounding inspection is time-consuming, and there is no convenient way to represent exhaustively the test results. The proposed approach solves these two problems by having an assistant robot following the inspector, and always being able to look at the hammer impact position. The assistant robot accurately estimates the position of the impact in real-time and creates a detailed representation of the test results. Experimental results show the process for creating the detailed inspection report. The accuracy of the human-robot cooperative approach is evaluated for a real world application. The average error of the impact point estimation was 32 millimeters and the standard deviation was 30 millimeters.

This paper presents a framework for creating a 3D map of an environment that contains the probability of a geometric feature to emit a sound. The goal is to provide an automated tool for condition monitoring of plants. The map is created by a mobile platform equipped with a microphone array and laser range sensors. The microphone array is used to estimate the sound power received from different directions whereas the laser range sensors are used for estimating the platform pose in the environment. During navigation, a ray casting method projects the audio measurements made onboard the mobile platform to the map of the environment. Experimental results show that the created map is an efficient tool for sound source localization.

This work introduces a 3D visibility model for comfortable autonomous vehicles. The model computes a visibility index based on the pose of the wheelchair within the environment. We correlate this index with human navigational comfort (discomfort) and we discuss the importance of modeling visibility to improve human riding comfort. The proposed approach models the 3D visual field of view combined with a two-layered environmental representation. The field of view is modeled with information from the pose of the robot, a 3D laser sensor and a two-layered environmental representation composed of a 3D geometric map with traversale area information. Human navigational discomfort was extracted from participants riding the autonomous wheelchair. Results show that there is fair correlation between poor visibility locations (e.g., blind corners) and human discomfort. The approach can model places with identical traversable characteristics but different visibility and it differentiates visibility characteristics according to traveling direction.

This paper presents a framework for making a mobile robot aware of an entity in the blind region of its laser range finders when that entity emits sound. First in a mapping stage, a 3D description of the environment that contains information about acoustic reflection is created. Then during operation, the robot combines estimated directions of arrival of sound with this 3D description to detect entities that are not visible by line of sight sensors but could be heard because of sound reflections. Using this approach, it is possible to restrict the hypothesis about the position of a sound emitting entity in the blind region to a small set of candidate depth values.

This work proposes a model for human habituation while riding a robotic wheelchair. We present and describe the concept of human navigational habituation which we define as the human habituation to repetitively riding a robotic wheelchair. The approach models habituation in terms of preferred linear velocity based on the experience of riding a wheelchair. We argue that preferred velocity changes as the human gets used to riding on the wheelchair. Inexperienced users initially prefer to ride at a slow moderate pace, however the longer they ride they prefer to speed up to a certain comfort level and find initial slower velocities to be tediously “too slow” for their experience level. The proposed habituation model provides passenger preferred velocity based on experience. Human biological measurements, galvanic skin conductance, and participant feedback demonstrate the preference for habituation velocity control over fixed velocity control. To our knowledge habituation modeling is new in the field of autonomous navigation and robotics.

This paper presents a multi-modal sensor approach for mapping sound sources using an omni-directional microphone array on an autonomous mobile robot. A fusion of audio data (from the microphone array), odometry information and the laser range scan data (from the robot) was used to precisely localize and map the audio sources in an environment. An audio map is created while the robot is autonomously navigating through the environment by continuously generating audio scans with a steered response power (SRP) algorithm. Using the poses of the robot, rays are cast in the map in all directions given by the SRP. Then each occupied cell in the geometric map hit by a ray is assigned a likelihood of containing a sound source. This likelihood is derived from the SRP at that particular instant. Since the localization of the robot is probabilistic, the uncertainty in the pose of the robot in the geometric map is propagated to the occupied cells hit during the ray casting. This process is repeated while the robot is in motion and the map is updated after every audio scan. The generated sound maps were reused and the changes in the audio environment were updated by the robot as it identifies these changes.

This paper presents a method for mapping the radiated sound intensity of an environment using an autonomous mobile platform. The sound intensities radiated by the objects are estimated by combining the sound intensity at the platform's position (estimated with a steered response power algorithm) and the distances to the objects (estimated using laser range finders). By combining the estimated sound intensity at the platform's position with the platform's pose obtained from a particle filter based localization algorithm, the sound intensity radiated from the objects is registered in the cells of a grid map covering the environment. This procedure creates a map of the radiated sound intensity that contains information about the sound directivity. To illustrate the effectiveness of the proposed method, a map of radiated sound intensity is created for a test environment. Then the position and the directivity of the sound sources in the test environment are estimated from this map.

We proposed a method for estimating sound source locations in a 3D space by integrating sound directions estimated by multiple microphone arrays and taking advantage of reflection information. Two types of sources with different directivity properties (human speech and loudspeaker speech) were evaluated for different positions and orientations. Experimental results showed the effectiveness of using reflection information, depending on the position and orientation of the sound sources relative to the array, walls, and the source type. The use of reflection information increased the source position detection rates by 10% on average and up to 60% for the best case.

This paper presents a method for detecting moving entities that are in the robot's path but not in the field of view of sensors like laser scanners, cameras or ultrasonic sensors. The proposed system makes use of passive acoustic localization methods which receive information from occluded regions (at intersections or corners) because of the multipath nature of sound propagation. Contrary to the conventional sensors, this method does not require line of sight. In particular, specular reflections in the environment make it possible to detect moving entities that emit sound such as a walking person or a rolling cart. This idea was exploited for safe navigation of a mobile platform at intersections. The passive acoustic localization output is combined with a 3D geometric map of the environment that is precise enough to estimate sound propagation and reflection using ray casting methods. This gives the robot the ability to detect a moving entity out of the field of view of the sensors that require line of sight. Then the robot is able to recalculate its path and waits until the detected entity is out of its path so that it is safe to move to its destination. To illustrate the performance of the proposed method, a comparison of the robot's navigation with and without the audio sensing is provided for several intersection scenarios.

Parkinson's disease (PD) is a severe disease with many symptoms, including speech disorders. Although many methods exist to treat some of PD's symptoms, therapies for speech impairment are not effective and satisfactory, resulting in an open area of research. The current project aims at taking advantage of the Lombard reflex to improve the speech loudness of PD patients. As a first step, the experience of the Lombard reflex by Japanese PD people was confirmed, and the perception of PD patients' speech was evaluated by several subjects. In a following step, methods based on masking sound will be used for intensive training and for self-training of PD patients. However, after intensive training, PD patients may be able to talk louder even without masking noise. In addition, the design and the development of a device based on masking sound that can be used by PD patients while using phone is under consideration.

This paper focuses on the problem of environmental noises in human-human communication and in automatic speech recognition. To deal with this problem, the use of alternative acoustic sensors -which are attached to the talker and receive the uttered speech through skin or bones- is investigated. In the current study, throat microphones and ear bone microphones are integrated with standard microphones using several fusion methods. The results obtained show that the recognition rates in noisy environments are drastically increased when these sensors are integrated with standard microphones. Moreover, the system does not show any recognition degradations in clean environments. In fact, recognition rates also increase slightly in clean environments. Using late fusion to integrate a throat microphone, an ear bone microphone, and a standard microphone, we achieved a 44% relative improvement in recognition rate in a noisy environment and a 24% relative improvement in recognition rate in a clean environment.

This paper presents an audio monitoring system for detecting and identifying people engaged in a conversation. The proposed method is hands-free as it uses a microphone array to acquire the sound. A particularity of the approach is the use of a laser range finder based human tracker system. The human tracker monitors the locations of people then local steered response power is used to detect the people speaking and localize precisely their mouths. Then an audio stream is created for each person and used to perform speaker identification. Experimental results show that the use of the human tracker has several benefits compared to an audio only approach.

This paper presents a novel method for solving the permutation problem inherent to frequency domain blind signal separation of multiple simultaneous speakers. As conventional methods, the proposed method exploits the direction of arrival (DOA) of the different speakers to resolve the permutation. But it is designed to exploit the information from pairs of microphones that are usually discarded because of the spatial aliasing. The proposed method is based on an explicit expression of the spatial aliasing effect on the DOA estimation. By introducing a vector of integer values in the equation used to estimate the DOA, it becomes possible to compensate the spatial aliasing by solving the equation relatively to that vector. The proposed method operates sequentially along the frequency bins. First the spatial aliasing is compensated by an iterative procedure that also detects the permutations. Then the detected permutation are suppressed and the DOA are estimated using all available pairs of microphones. Some simulation results demonstrate the effectiveness of the method.

Small informal meetings of two to four participants are very common in work environments. For this reason, a convenient way for recording and archiving these meetings is of great interest. In order to efficiently archive such meetings, an important task to address is to keep trace of “who talked when” during a meeting. This paper proposes a new multi-modal approach to tackle this speaker activity detection problem. One of the novelty of the proposed approach is that it uses a human tracker that relies on scanning laser range finders (LRFs) to localize the participants. This choice is especially relevant for robotic applications as robots are often equipped with LRFs for navigation purpose. In the proposed system, a table top microphone array in the center of the meeting room acquires the audio data while the LRF based human tracker monitors the movement of the participants. Then the speaker activity detection is performed using Gaussian mixture models that were trained before hand. An experiment reproducing a meeting configuration demonstrates the performance of the system for speaker activity detection. In particular, the proposed hands free system maintains an good level of performance compared to the use of close talking microphone while participants are simultaneously speaking.

Several recent methods for speech enhancement in presence of diffuse background noise use frequency domain blind signal separation to estimate the diffuse noise and a nonlinear post filter to suppress this estimated noise. This paper presents a frequency domain blind signal extraction method for estimating the diffuse noise in place of the frequency domain blind signal separation. The method is based on the minimization by means of a complex Newton algorithm of a cost function depending of the modulus of the extracted component. The proposed complex Newton method is compared to the gradient descent on the same cost function and to the blind signal separation approach.

This paper study the blind estimation of the diffuse background noise for the hands-free speech interface. Some recent papers showed that it is possible to use blind signal separation (BSS) to estimate the diffuse background noise by suppressing the speech component after all the components were separated. In particular, the scale indeterminacy of BSS is avoided by using the projection back method. In this paper, we study an alternative to the projection back for the noise estimation and justify the use of blind signal extraction BSE rather than BSS.

This paper presents a noise cancellation method based on the ability to efficiently cancel a close target speaker contribution from the signals observed at a microphone array. The proposed method exploits this specificity in the case of the hands-free speech interface. This method is in particular able to deal with non-stationary noise. The method can be divided in three steps. First, the steering vector pointing at the target user is estimated from the covariance of the observed signals. Then the noise estimate is obtained by cancelling the user's contribution. During this step the speech pauses are also estimated. Finally a post-filter is used to suppress this estimated noise from the observed signals. The post-filter strength is controlled by using the estimated noise during the speech pauses as reference. A 20k-words dictation task in presence of non-stationary diffuse background noise at different SNR levels illustrates the effectiveness of the proposed method.

In this paper, we propose a microphone array structure for a spoken-oriented robot dialog system that is designed to discriminate the direction of arrival (DOA) of the target speech and that of the robot internal noise. First, we investigate the performance of the noise estimation conducted by semi-blind source separation (SBSS) in presence of both the diffuse background noise and the robot internal noise. The result indicates that the noise estimation of the SBSS is not good. Next, we analyze the DOA of the robot internal noise in order to determine the reason of the above result; we find out that the internal noise is always in-phase at the microphone array and overlap spacial with the target speech. Based on this fact, we propose to change the microphone array structure from the broadside array to the end-fire array in order to discriminate the DOAs of the target speech and the internal noise. Finally, we evaluate the word accuracy in a dictation task in presence of both diffuse background noise and robot internal noise to confirm the advantage of the proposed structure. Simulation results shows that the proposed microphone array structure results in approximately 10% improvement of the speech recognition performance.

The speech enhancement architecture presented in this paper is specifically developed for hands-free robot spoken dialog systems. It is designed to take advantage of additional sensors installed inside the robot to record the internal noises. First a modified frequency domain blind signal separation (FD-BSS) gives estimates of the noises generated outside and inside of the robot. Then these noises are canceled from the acquired speech by a multichannel Wiener post-filter. Some experimental results show the recognition improvement for a dictation task in presence of both diffuse background noise and internal noises.

Recently, methods using blind signal separation were proposed to separate the signals received by a microphone array. In this paper, we propose a new frequency domain semi-blind source separation method for replacing the blind source separation method when it is possible to obtain additional information on some of the signals. This is of particular interest in situations like in hands-free speech recognition where the blind separation has to work on limited amount of data in a challenging environment. The proposed method incorporates references to some of the signals that are obtained by additional sensors. Some experimental results shows that the proposed method is able to incorporate the additional information efficiently and that the performances are improved in term of SNR and word accuracy in a speech recognition task.

We propose a robust and fast dereverberation technique for real-time speech recognition application. First, we effectively identify the late reflection components of the room impulse response. We use this information together with the concept of Spectral Subtraction (SS) to remove the late reflection components of the reverberant signal. In the absence of the clean speech in actual scenario, approximation is carried out in estimating the late reflection where the estimation error is corrected through multi-band SS. The multi-band coefficients are optimized during offiine training and used in the actual online dereverberation. The proposed method performs better and faster than the relevant approach using Multi-LPC and reverberant matched model. Moreover the proposed method is robust to speaker and microphone locations.

INTERSPEECH2008: 9th Annual Conference of the International Speech Communication Association, September 22-26, 2008, Brisbane, Australia.

The model of the human/machine hands-free speech interface is defined as a point source (the user voice) and a diffuse background noise. This situation is very different from the usual cocktail party model, separation of a mixture of speeches, that is usually treated in frequency domain blind signal separation (FD-BSS). In particular, the fast permutation solvers proposed for the cocktail party model results in poor separation performance in this case. In order to resolve the permutation more efficiently, this paper proposes a new approach that exploits the statistical discrepancy between the target speech and the diffuse background noise.