In applications like noise cancellation and virtual reality, precise sound source localization is crucial. Existing data-driven binaural systems offer high performance in adverse conditions such as noise and reverberation but face limitations with real-time operation and performance degradation in HRTF mismatch scenarios. Our work introduces a compact Vision Transformer tailored to address these issues, with a primary focus on horizontal speech localization. Inspired by the auditory cortex, our model uniquely incorporates spectral attention mechanisms using encoded speech representations. This architecture enhances generalization on the azimuth plane under mismatched HRTFs. Our empirical results show a marked improvement over conventional DNN, CNN-based and Transformer-based models, both in noisy and noise-free environments. Significantly, the proposed model maintains high accuracy in localizing adjacent azimuths, ideal for real-world applications.

International audience

This paper presents non-negative factorization of audio signals for the binaural localization of multiple sound sources within realistic and unknown sound environments. Non-negative tensor factorization (NTF) provides a sparse representation of multichannel audio signals in time, frequency, and space that can be exploited in computational audio scene analysis and robot audition for the separation and localization of sound sources. In the proposed formulation, each sound source is represented by means of spectral dictionaries, temporal activation, and its distribution within each channel (here, left and right ears). This distribution, being dependent on the frequency, can be interpreted as an explicit estimation of the Head-Related Transfer Function (HRTF) of a binaural head which can then be converted into the estimated sound source position. Moreover, the semisupervised formulation of the non-negative factorization allows us to integrate prior knowledge about some sound sources of interest whose dictionaries can be learned in advance, whereas the remaining sources are considered as background sound, which remains unknown and is estimated on the fly. The proposed NTF-based sound source localization is applied here to binaural sound source localization of multiple speakers within realistic sound environments.

This paper deals with speaker localization in two dimensions from a mobile binaural head. A bootstrap particle filtering scheme is used to perform active localization, i.e. to infer source location by fusing the binaural perception with the sensor motor commands. It relies on an original pseudo-likelihood of the source azimuth which captures both the interaural level and phase differences. Since the pseudo-likelihood is discrete, it is fitted with a mixture of circular distributions in order to enhance its resolution. For the fitting task two mixtures are compared and evaluated, namely the mixture of von Mises and wrapped Cauchy distributions. Furthermore, a solution is presented for calculating the von Mises curvefitting with low uncertainty, since the direct implementation can quickly surpass double precision floating number representation. The performance of the filter is compared using both the raw and fitted pseudo-likelihoods on experiments recorded in an acoustically prepared room with ground-truth obtained from a motion capture system. The results show that the proposed algorithm successfully localizes the speaker with an advantage in the direction of the fitted von Mises mixture likelihood.

A theoretically grounded scheme to Direction of Arrival (DOA) estimation and Source Activity Detection (SAD) is proposed, on the basis of a pair of microphones. The method can capture the effects of the robot's scatterers, if any. The DOA estimator takes place within a probabilistic framework and outputs the Maximum Likelihood Estimate (MLE) of the DOA with respect to the collected audio data. Besides, the SAD relies on statistical identification. The behavior of the estimator is studied under various operating modes, considering free-field propagation and scattering by a rigid spherical head. Experimental results validate the approach.

This paper proposes a sound localization algorithm inspired by a cross-channel algorithm first studied by MacDonald et. al in 2008. The original algorithm assumes that the Head Related Transfer Functions (HRTFs) of the robotic head under study are precisely known, which is rarely the case in practice. Following the idea that any head is more or less spherical, the above assumption is relaxed by using HRTFs computed using a simple spherical head model with the same head radius as the robot head. In order to evaluate the proposed approach in realistic noisy conditions, an isotropic noise field is also computed and a precise definition of the Signal to Noise Ratio (SNR) in a binaural context is outlined. All these theoretical developments are finally assessed with simulated and experimental signals. Despite its simplicity, the proposed approach appears to be robust to noise and to provide reliable sound localization estimations in the frontal azimuthal plane.

Sound source localization is an important feature designed and implemented on robots and intelligent systems. Like other artificial audition tasks, it is constrained to multiple problems, notably sound reflections and noises. This paper presents a sound source azimuth estimation approach in reverberant environments. It exploits binaural signals in a humanoid robotic context. Interaural Time and Level Differences (ITD and ILD) are extracted on multiple frequency bands and combined with a neural network-based learning scheme. A cue filtering process is used to reduce the reverberations effects. The system has been evaluated with simulation and real data, in multiple aspects covering realistic robot operating conditions, and was proven satisfying and effective as will be shown and discussed in the paper.

The goal of this work is to build an effective and practical system to diagnose and prognose faults of complex systems, like aircraft, satellite and so on. In this paper, a machine-learning method Gaussian Mixture Models (GMMs) is used to automatically detect, isolate, and even forecast the faults, while keeping the reliability and safety of complex system. Each dysfunctional model is completed by GMMs during machine learning, which constitutes the diagnosis system to distinguish and troubleshooting the faults. On the other side, principal component analysis (PCA) is combined with the system to improve the efficiency of GMMs, which can effectively compress the high dimensional data. Except for that, GMMs helps the system to achieve the visualization of dysfunctional models. With this visualization, the prognosis system can surveil the evolution of data and estimate their tendency, which is important to forecast the next condition of the complex system. The diagnosis and prognosis system proposed in this paper has been fully tested by using actual experimental data of aircraft X, which is supplied by Dassault Aviation.

A fundamental task for a robotic audition system is sound source localization. This paper addresses the localization problem in a robotic humanoid context, providing a novel learning algorithm that uses binaural cues to determine the sound source's position. Sound signals are extracted from a humanoid robot's ears. Binaural cues are then computed to provide inputs for a neural network. The neural network uses pixel coordinates of a sound source in a camera image as outputs. This learning approach provides good localization performances as it reaches very small errors for azimuth and elevation angles estimates.

Developmental Robotics offers a new approach to numerous AI features that are often taken as granted. Traditionally, perception is supposed to be an inherent capacity of the agent. Moreover, it largely relies on models built by the system's designer. A new approach is to consider perception as an experimentally acquired ability that is learned exclusively through the analysis of the agent's sensorimotor flow. Previous works, based on H.Poincaré's intuitions and the sensorimotor contingencies theory, allow a simulated agent to extract the dimension of geometrical space in which it is immersed without any a priori knowledge. Those results are limited to infinitesimal movement's amplitude of the system. In this paper, a non-linear dimension estimation method is proposed to push back this limitation.

This paper takes place within the field of active sound source localization in a binaural context. A stochastic filtering strategy is presented for the localization of a still or moving source from a moving binaural sensor. The proposed method accounts for the source intermittence as well as for false measurements induced by the non-stationarity of the emitted signal. Its effectiveness is showed on experimental results.

Sound source localization is a need for robotic systems interacting with acoustically-active environments. In this domain, numerous binaural localization studies have been conducted within the last few decades. This paper provides an overview of a number of binaural localization cue extraction techniques. These are carefully addressed and applied on a simulated binaural database. Cues are evaluated in azimuth estimation and their discriminatory effectiveness is studied as a function of the reverberation time with statistical data analysis techniques. Results show that big differences exist between the discriminatory abilities of multiple types of cue extraction methods. Thus a careful cue selection must be performed before establishing a sound localization system.

This paper deals with binaural sound localization. An active strategy is proposed, relying on a precise model of the dynamic changes induced by motion on the auditive perception. The proposed framework allows motions of both the sound source and the sensor. The resulting stochastic discrete-time model is then exploited together with Unscented Kalman filtering to provide range and azimuth estimation. Simulations and experiments show the effectiveness of the method.

In this paper, an original study of a binaural speaker identification system is presented. The state of the art shows that, contrarily to monaural and multi-microphone approaches, binaural systems are not so much studied in the specific task of automatic speaker recognition. Indeed, these systems are mostly used for speech recognition, or speaker localization. This study will focus on the benefits of the binaural context in comparison with monaural techniques. It demonstrates the interest of the binaural systems typically used in humanoid robotics. The system is first tested with monaural signals, and then with a binaural sensor, in many signal to noise ratios, speech durations and speaker directions. Up to 11 percent of improvement in recognition ratios of 23 ms frames can be obtained. The used database is a set of audio tracks recorded for 10 speakers, and filtered by HRTFs to obtain binaural signals in the directions of interest, for the binaural training and testing steps. This way, we study the sensitivity of the system to the speaker's location in an environment where a maximum of 10 speakers is present.

Perception and action are fundamental tasks for autonomous robots. Traditionnally, they rely on theoretical models built by the system's designer. But, is a naive agent able to learn by itself the structure of its interaction with the environment without any a priori information? This knowledge should be extracted through the analysis of the only information it has access to: its high-dimensional sensorimotor flow. Recent works, based on the sensorimotor contingencies theory, allow a simulated agent to extract the geometrical space dimensionality without any model of itself nor of the environment. In this paper, these results are validated using a more sophisticated auditive modality. The question of multimodality fusion is then addressed by fitting up the agent with vision. Finally, preliminary experimental results on a real robotic platform are presented.

This paper presents the embedded audition for robotics (EAR) project internally developed at LAAS and its application to speaker localization and extraction. Hardware and software issues are first thoroughly depicted, concerning the development of an auditory sensor based on an array of microphones, a homemade dedicated acquisition chain and a FPGA based processing board. Then, the EAR sensor is assessed against various scenarios, in real noisy robotics environments. Localization results are presented when a speaker emits an utterance in the presence of a disturbing source. These validate the underlying theory and suggest further theoretical and experimental developments.

The MUSIC algorithm (multiple signal classification) is a well-known high-resolution method to sound source localization. However, as it is essentially narrowband, several extensions can be envisaged when dealing with broadband sources like human voice. This paper presents such extensions and proposes a comparative study w.r.t. specific robotics constraints. An online beamspace MUSIC method, together with a recently developed beamforming scheme, are shown to constitute a mathematically sound and potentially efficient solution.

This paper describes a broadband beampattern synthesis method for sound source localization in the nearfield or in the farfield of a mobile robot, with a small-size linear array. The method is based on the theory of modal analysis and involves an original convex optimization procedure which benefits from the Parseval relation. The optimized beampattern is obtained by numerically minimizing the worst-case error between the modal coefficients of the array response and those of the reference beampattern, up to a finite rank of the series expansion, over a frequency grid. Simulations illustrate the analytical development

The work presented in this paper comes as a part of a project which aims at developing an auditory system for a mobile robot. It presents a sound source localization strategy which enables the sensing of signals within a direction of arrival and frequency domain of interest while rejecting other data. A rapid prototyping method is proposed to design filter-sum beamformers on the basis of convex optimization. This method is well-suited to robotics applications as it copes with real-time constraints and allows the localization of broadband signals such as human voice. Numerous simulation results are used to illustrate the reasoning.

This paper addresses the problem of practically implementing an original sound source localization strategy for mobile robots applications. The proposed method is based on a convex optimization solution to beamforming. It allows the sensing of signals within a direction of arrival and frequency domain of interest while rejecting other data. A precise description of the acquisition chain is proposed and a careful mathematical modeling is given in order to bridge the gap between theory and practical implementation. Simulation results and comparisons with classical filter-sum beamformer techniques are provided at the end of the paper to illustrate the performance of the sensor.