The ability to generalize well to different environments is of importance for audio de-noising systems in real-world scenarios. Especially single-channel signals require efficient noise filtering without impacting speech intelligibility negatively. Our previous work has shown that a probabilistic latent space model combined with a U-Network architecture increases performance and generalization ability to some extent. Here, we further evaluate magnitude-only, as well as complex-valued U-Network models, on two different datasets, and in a train-test mismatch scenario. Adaptability of models is evaluated by introducing a curve-based score similar to area-under-the-curve metrics. The proposed probabilistic latent space models outperform their ablated variants in most conditions, as well as well-known comparison methods, while increases in network size are negligible. Improvements of up to 0.97 dB SI-SDR in matched, and 2.72 dB SI-SDR in mismatched conditions are observed, with highest total SI-SDR scores of 20.21 dB and 18.71 dB, respectively. The proposed stability-score aligns well with observed performance behaviour, further validating the probabilistic latent space model.

In this paper, we propose to extend the deep, complex U-Network architecture for speech enhancement by incorporating a probabilistic (i.e., variational) latent space model. The proposed model is evaluated against several ablated versions of itself in order to study the effects of the variational latent space model, complex-value processing, and self-attention. Evaluation on the MS-DNS 2020 and Voicebank+Demand datasets yields consistently high performance. E.g., the proposed model achieves an SI-SDR of up to 20.2 dB, about 0.5 to 1.4 dB higher than its ablated version without probabilistic latent space, 2–2.4 dB higher than WaveUNet, and 6.7 dB above PHASEN. Compared to real-valued magnitude spectrogram processing with a variational U-Net, the complex U-Net achieves an improvement of up to 4.5 dB SI-SDR. Complex spectrum encoding as magnitude and phase yields best performance in anechoic conditions whereas real and imaginary part representation results in better generalization to (novel) reverberation conditions, possibly due to the underlying physics of sound.

Classic approaches to multi-channel signal enhancement rely on model assumptions regarding speech source relative transfer functions and noise covariance matrix, or on estimates thereof obtained in, e.g., speech pauses. To alleviate these constraints, we here investigate an approach to adaptive estimation of the speech (target) source and noise related acoustic parameters based on localized speech probability estimates. The latter are computed from a discriminatively trained speech localization algorithm previously proposed [1]. A-priori knowledge of temporal segments that contain noise only, thus, is not required. A standard MVDR system is employed for subsequent signal enhancement. Evaluation is carried out for anechoic and reverberant conditions using 6-channel input signals recorded with a bilateral hearing-aid geometry. Results indicate that the proposed method outperforms an anechoic, isotropic-noise model when a-priori information is unavailable: I.e., in (a) anechoic conditions with localized interferer in addition to isotropic noise, and (b) reverberant conditions. In these conditions, the proposed method and constrained versions thereof improve upon the free-field isotropic noise model by up to 16.1 dB and 7.7 dB SINR, respectively.

This paper evaluates neural network (NN) based systems and compares them to Gaussian mixture model (GMM) and hidden Markov model (HMM) approaches for acoustic scene classification (SC) and polyphonic acoustic event detection (AED) that are applied to data of the “Detection and Classification of Acoustic Scenes and Events 2016” (DCASE'16) challenge, task 1 and task 3, respectively. For both tasks, the use of deep neural networks (DNNs) and features based on an amplitude modulation filterbank and a Gabor filterbank (GFB) are evaluated and compared to standard approaches. For SC, additionally a time-delay NN approach is proposed that enables analysis of long contextual information similar to recurrent NNs but with training efforts comparable to conventional DNNs. The SC system proposed for task 1 of the DCASE'16 challenge attains a recognition accuracy of 77.5%, which is 5.6% higher compared to the DCASE'16 baseline system. For the AED task, DNNs are adopted in tandem and hybrid approaches, i.e., as part of HMM-based systems. These systems are evaluated for the polyphonic data of task 3 from the DCASE'16 challenge. Several strategies to address the issue of polyphony are considered. It is shown that DNN-based systems perform less accurate than the traditional systems for this task. Best results are achieved using GFB features in combination with a multiclass GMM-HMM back end.

This contribution investigates the use of features derived from a Gabor filterbank (GFB) for the application of acoustic cough classification. Gabor filters are two-dimensional filters that decompose the spectro-temporal power density further into components which capture spectral, temporal and joint spectro-temporal modulation patterns. The proposed GFB feature extraction scheme in combination with Gaussian mixture model (GMM) and hidden Markov model (HMM) classifier back-ends is evaluated using a cough database recorded by a phone hotline. The database is composed of two kind of coughs, i.e., dry and productive cough, and other sounds, e.g. speech. Based on these data, we show that GFB features result in better recognition performance than the common Mel-frequency cepstral coefficient (MFCC) baseline for the given task of cough classification. Furthermore, results indicate that GMMs are preferable to HMMs for this kind of data.

S.2562-2566

Inspired by physiological studies on the human auditory system and by results from psychoacoustics, an amplitude modulation filter bank (AMFB) has been developed and successfully applied to feature extraction for automatic speech recognition (ASR) in earlier work. Here, we address the question as to which amplitude modulation (AM) frequency decomposition leads to optimal ASR performance by proposing a parameterized functional relationship between modulation center frequency and modulation bandwidth. Word error rates (WERs) of ASR experiments with 1551 different AMFBs are systematically evaluated and compared, resulting in the identification of a comparatively narrow range of optimal modulation frequency to modulation bandwidth characteristics. To integrate modulation processing with deep neural network (DNN) acoustic modeling, we propose merging of modulation filter coefficients with DNN weights prior to a final training step and an improved mean-variance normalization scheme for AMFBs. These modifications are shown to result in further reduction of WERs and are indicative of the proposed system's improved generalization ability, when compared across corpora of 100-960 h of data with mismatched training and test conditions. Analysis of DNN-learned temporal AM filtering properties is carried out and implications for the relevance of different modulation regions as well as the relation to psychoacoustic findings are discussed. ASR experiments with the proposed system demonstrate a high degree of robustness against extrinsic acoustic distortions, resulting in, e.g., an average WER of 9.79% on the Aurora-4 task.

The paper describes an automatic speech recognition (ASR) system for the 3rd CHiME challenge that addresses noisy acoustic scenes within public environments. The proposed system includes a multi-channel speech enhancement front-end including a microphone channel failure detection method that is based on cross-comparing the modulation spectra of speech to detect erroneous microphone recordings. The main focus of the submission is the investigation of the amplitude modulation filter bank (AMFB) as a method to extract long-term acoustic cues prior to a Gaussian mixture model (GMM) or deep neural network (DNN) based ASR classifier. It is shown that AMFB features outperform the commonly used frame splicing technique of filter bank features even on a performance optimized ASR challenge system. I.e., temporal analysis of speech by hand-crafted and auditory motivated AMFBs is shown to be more robust compared to a data-driven method based on extracting temporal dynamics with a DNN. Our final ASR system, which additionally includes adaptation of acoustic features to speaker characteristics, achieves an absolute word error rate reduction of approx. 21.53 % relative to the best CHiME-3 baseline system on the "real" test condition.

The human ability to classify acoustic sounds is still unmatched compared to recent methods in machine learning. Psychoacoustic and physiological studies indicate that the auditory system of mammals decomposes audio signals into their acoustic and modulation frequency components prior to further analysis. Since it is known that most linguistic information is coded in amplitude fluctuations, mimicking temporal processing strategies of the auditory system in automatic speech recognition (ASR) promises to increase recognition accuracies. We present an amplitude modulation filter bank (AMFB) that is used as a feature extraction scheme in ASR systems. The time-frequency resolution of the employed FIR filters, i.e., bandwidth and modulation frequency settings, are adopted from a psychophysically inspired model of Dau (1997) that was originally proposed to describe data from human psychoacoustics. Investigations on modulation phase indicate the need for preserving such information in amplitude modulation features. We show that the filter symmetry has an important impact on ASR performance. The proposed feature extraction scheme exhibits significant word error rate (WER) reductions using the Aurora-2, Aurora-4, and REVERB ASR tasks compared to other recent feature extraction methods, such as MFCC, FDLP, and PNCC features. Thereby, AMFB features reveal high robustness against additive noise, different transmission channel characteristics, and room reverberation. Using the Aurora-4 benchmark, for instance, an average WER of 12.33% with raw and 11.31% with bottleneck transformed features is attained, which constitutes a relative improvement of 19.6% and 29.2% over raw MFCC features, respectively.

Algorithms for the automatic detection and recognition of acoustic events are increasingly gaining relevance for the reliable and robust functioning of consumer, assistive and monitoring systems. The extraction of appropriate task relevant acoustic features from the raw sound signal clearly influences performance of subsequent statistical classification, in particular in adverse acoustic situations. The present contribution investigates the use of biologically-inspired features, derived from a filterbank of two-dimensional Gabor functions, that decompose the spectro-temporal power density into components which capture spectral, temporal and joint spectro-temporal modulation patterns. It is hypothesized that the comparably large joint spectral and temporal extent of these Gabor functions results in features that allow for robust classification. Evaluation of the proposed feature extraction scheme together with an hidden Markov model (HMM) classifier is conducted on two corpora comprising acoustic events in realistic adverse conditions from the D-CASE and CLEAR'07 evaluation campaigns. Relevance of each Gabor filter for classification is analyzed and an optimized parameter set for the Gabor filterbank (GFB) is identified. Performance of the optimized GFB is evaluated in comparison to other state-of-the-art algorithms on isolated event classification and on the full acoustic event detection (AED) including joint classification and temporal segmentation of events. Results show that Gabor features result in a signal representation that exhibits separated average class-specific patterns. An improvement in classification accuracy of up to 26% relative to the Mel-frequency cepstral coefficient (MFCC) baseline is obtained with the optimized GFB. Further experiments demonstrate that this improvement cannot be explained by purely temporal or purely spectral Gabor basis functions. Rather, a GFB with features extending in joint spectro-temporal directions is required to obtain optimum performance. Performance on AED with the D-CASE challenge dataset is shown to improve on previous algorithms from the recent literature.

State-of-the-art classifiers like hidden Markov models (HMMs) in combination with mel-frequency cepstral coefficients (MFCCs) are flexible in time but rigid in the spectral dimension. In contrast, part-based models (PBMs) originally proposed in computer vision consist of parts in a fully deformable configuration. The present contribution proposes to employ PBMs in the spectro-temporal domain for detection of emergency siren sounds in traffic noise,standard generative training resulting in a classifier that is robust to shifts in frequency induced, e.g., by Doppler-shift effects. Two improvements over standard machine learning techniques for PBM estimation are proposed: (i) Spectro-temporal part (“appearance”) extraction is initialized by interest point detection instead of random initialization and (ii) a discriminative training approach in addition to standard generative training is implemented. Evaluation with self-recorded police sirens and traffic noise gathered on-line demonstrates that PBMs are successful in acoustic siren detection. One hand-labeled and two machine learned PBMs are compared to standard HMMs employing mel-spectrograms and MFCCs in clean and multi condition (multiple SNR) training settings. Results show that in clean condition training, hand-labeled PBMs and HMMs outperform machine-learned PBMs already for test data with moderate additive noise. In multi condition training, the machine learned PBMs outperform HMMs on most SNRs, achieving high accuracies and being nearly optimal up to 5 dB SNR. Thus, our simulation results show that PBMs are a promising approach for acoustic event detection (AED).

Physiologically inspired feature extraction for audio classification often uses simplified parametric models of auditory processing. We employ linear and nonlinear neuron models directly derived from neural responses in zebra finches as feature extraction front-ends. The most important features were identified using automatic feature selection techniques. This allows both a quantitative evaluation of neural features for sound classification tasks in terms of classification accuracy and a qualitative analysis of the auditory features that are most relevant. It turned out that a relatively small subpopulation of neural responses is sufficient to achieve reasonable classification performance. For linear as well as for nonlinear neuron models, we found three different shapes of spectro-temporal features to be archetypical. The relation of these to analytic approaches (such as Gabor filters) is discussed. The overall classification rates in a 6-class task reached up to 94% accuracy. Nonlinear models provided up to 15% benefit over linear models, indicating the importance of nonlinearities in classification with physiologically motivated features.

Many research efforts in the field of feature extraction for automatic speech recognition are focused on analyzing slow amplitude fluctuations of speech. In this study the importance of spectral and temporal resolution for the amplitude modulation frequency analysis are investigated in order to provide guidance for the appropriate filter design. Therefore, different wavelet and Fourier transform like filter time scales are examined, i.e. The importance of time and frequency separation is compared. The results demonstrate that analyzing three separate amplitude modulation frequency bands of constant absolute bandwidth that cover the range from about 2 to 16 Hz are sufficient for automatic speech recognition.

Unexpected stimuli are a challenge to any machine learning algorithm. Here, we identify distinct types of unexpected events when general-level and specific-level classifiers give conflicting predictions. We define a formal framework for the representation and processing of incongruent events: Starting from the notion of label hierarchy, we show how partial order on labels can be deduced from such hierarchies. For each event, we compute its probability in different ways, based on adjacent levels in the label hierarchy. An incongruent event is an event where the probability computed based on some more specific level is much smaller than the probability computed based on some more general level, leading to conflicting predictions. Algorithms are derived to detect incongruent events from different types of hierarchies, different applications, and a variety of data types. We present promising results for the detection of novel visual and audio objects, and new patterns of motion in video. We also discuss the detection of Out-Of- Vocabulary words in speech recognition, and the detection of incongruent events in a multimodal audiovisual scenario.

In this contribution we present a feature extraction method that relies on the modulation-spectral analysis of amplitude fluctuations within sub-bands of the acoustic spectrum by a STFT. The experimental results indicate that the optimal temporal filter extension for amplitude modulation analysis is around 310 ms. It is also demonstrated that the phase information of the modulation spectrum contains important cues for speech recognition. In this context, the advantage of an odd analysis basis function is considered. The best presented features reached a total relative improvement of 53.5% for clean-condition training on Aurora-2. Furthermore, it is shown that modulation features are more robust against room reverberation than conventional cepstral and dynamic features and that they strongly benefit from a high early-to-late energy ratio of the characteristic RIR.

We tackle the task of localizing speech signals on the horizontal plane using monaural cues. We show that monaural cues as incorporated in speech are efficiently captured by amplitude modulation spectra patterns. We demonstrate that by using these patterns, a linear Support Vector Machine can use directionality related information to learn to discriminate and classify sound location at high resolution. We propose a straightforward and robust way of integrating information from two ears: treating each ear as an independent processor and integrate the information at the decision level by doing that ambiguity is to a large extent resolved.

Robust detection of speech embedded in real acoustic background noise is considered using an approach based on subband amplitude modulation spectral (AMS) features and trained discriminative classifiers. Performance is evaluated in particular for situations in which speech is embedded in acoustic backgrounds not presented during classifier training, and for signal-to-noise ratios (SNR) from -10 dB to 20 dB. The results show that (1) Generalization to novel background classes with AMS features yields better performance in 84% of investigated situations, corresponding to an SNR benefit of about 10 dB compared to mel-frequency cepstral coefficient (MFCC) features. (2) On known backgrounds, AMS and MFCCs achieve similar performance, with a small advantage for AMS in negative SNR regimes. (3) Standard voice activity detection (ITU G729.B) performs significantly worse than the classification-based approach.

In this study, a new generic framework for the detection and interpretation of disagreement (“incongruence”) between different classifiers [1] is applied to the problem of detecting novel acoustic objects in an office environment. Using a general model that detects generic acoustic objects (standing out from a stationary background) and specific models tuned to particular sounds expected in the office, a novel object is detected as an incongruence between the models: the general model detects it as a generic object, but the specific models can not identify it as any of the known office-related sources. The detectors are realized using amplitude modulation spectrogram and RASTA-PLP features with support vector machine classification. Data considered are speech and non-speech sounds embedded in real office background at signal-to-noise ratios (SNR) from +20 dB to −20 dB. Our approach yields approximately 90% hit rate for novel events at 20 dB SNR, 75% at 0 dB and reaches chance level below −10 dB.

It is of prime importance in everyday human life to cope with and respond appropriately to events that are not foreseen by prior experience. Machines to a large extent lack the ability to respond appropriately to such inputs. An important class of unexpected events is defined by incongruent combinations of inputs from different modalities and therefore multimodal information provides a crucial cue for the identification of such events, e.g., the sound of a voice is being heard while the person in the field-of-view does not move her lips. In the project DIRAC ("Detection and Identification of Rare Audio-visual Cues") we have been developing algorithmic approaches to the detection of such events, as well as an experimental hardware platform to test it. An audio-visual platform ("AWEAR" - audio-visual wearable device) has been constructed with the goal to help users with disabilities or a high cognitive load to deal with unexpected events. Key hardware components include stereo panoramic vision sensors and 6-channel worn-behind-the-ear (hearing aid) microphone arrays. Data have been recorded to study audio-visual tracking, a/v scene/object classification and a/v detection of incongruencies.

A classification method is presented that detects the presence of speech embedded in a real acoustic background of non-speech sounds. Features used for classification are modulation components extracted by computation of the amplitude modulation spectrogram. Feature selection techniques and support vector classification are employed to identify modulation components most salient for the classification task and therefore considered as highly characteristic for speech. Results show that reliable detection of speech can be performed with less than 10 optimally selected modulation features, the most important ones are located in the modulation frequency range below 10 Hz. Detection of speech in a background of non-speech signals is performed with about 90% test-data accuracy at a signal-to-noise level of 0 dB. Compared to standard ITU G729.B voice activity detection, the proposed method results in increased true positive and reduced false positive rates induced by a real acoustic background.

This paper introduces the new OLdenburg LOgatome speech corpus (OLLO) and outlines design considerations during its creation. OLLO is distinct from previous ASR corpora as it specifically targets (1) the fair comparison between human and machine speech recognition performance, and (2) the realistic representation of intrinsic variabilities in speech that are significant for automatic speech recognition (ASR) systems. To enable an unbiased human-machine comparison, OLLO is designed for recognition of individual phonemes that are embedded in logatomes, specifically, three-phoneme sequences with no semantic information. A balanced set of target-phonemes important for human and automatic speech recognition has been chosen, drawing on pilot ASR studies and cross-fertilization from the field of human speech intelligibility testing. Several intrinsic variabilities in speech are represented in OLLO, by recording from 40 speakers from four German dialect regions, and by covering six articulation characteristics. Results from preliminary phonetic time-labeling and ASR experiments are promising and consistent with corpus variabilities. 1.