State-of-the-art hearing aids (HA) are limited in recognizing acoustic environments. Much effort is spent on research to improve listening experience for HA users in every acoustic situation. There is, however, no dedicated public database to train acoustic environment recognition algorithms with a specific focus on HA applications accounting for their requirements. Existing acoustic scene classification databases are inappropriate for HA signal processing. In this work we propose a novel, binaural HA acoustic environment recognition data set (HEAR-DS) suitable for the environment recognition needs of HAs. We present the details about each individual environment provided within the data set. To show separability of these acoustic environments we trained a group of deep neural network-based classifiers which vary in complexity. The obtained classification accuracies provide a reliable indicator about the validity and separability of the provided data set. Finally, as we do not aim at providing the best possible neural network architecture to perform such a classification, but propose solely a novel data set, further research is needed to streamline such networks and optimize them for robustness, real-time and limited computational capability to fit into modern HAs.

Physiologically inspired feature extraction for audio classification often uses simplified parametric models of auditory processing. We employ linear and nonlinear neuron models directly derived from neural responses in zebra finches as feature extraction front-ends. The most important features were identified using automatic feature selection techniques. This allows both a quantitative evaluation of neural features for sound classification tasks in terms of classification accuracy and a qualitative analysis of the auditory features that are most relevant. It turned out that a relatively small subpopulation of neural responses is sufficient to achieve reasonable classification performance. For linear as well as for nonlinear neuron models, we found three different shapes of spectro-temporal features to be archetypical. The relation of these to analytic approaches (such as Gabor filters) is discussed. The overall classification rates in a 6-class task reached up to 94% accuracy. Nonlinear models provided up to 15% benefit over linear models, indicating the importance of nonlinearities in classification with physiologically motivated features.

Unexpected stimuli are a challenge to any machine learning algorithm. Here, we identify distinct types of unexpected events when general-level and specific-level classifiers give conflicting predictions. We define a formal framework for the representation and processing of incongruent events: Starting from the notion of label hierarchy, we show how partial order on labels can be deduced from such hierarchies. For each event, we compute its probability in different ways, based on adjacent levels in the label hierarchy. An incongruent event is an event where the probability computed based on some more specific level is much smaller than the probability computed based on some more general level, leading to conflicting predictions. Algorithms are derived to detect incongruent events from different types of hierarchies, different applications, and a variety of data types. We present promising results for the detection of novel visual and audio objects, and new patterns of motion in video. We also discuss the detection of Out-Of- Vocabulary words in speech recognition, and the detection of incongruent events in a multimodal audiovisual scenario.

Robust detection of speech embedded in real acoustic background noise is considered using an approach based on subband amplitude modulation spectral (AMS) features and trained discriminative classifiers. Performance is evaluated in particular for situations in which speech is embedded in acoustic backgrounds not presented during classifier training, and for signal-to-noise ratios (SNR) from -10 dB to 20 dB. The results show that (1) Generalization to novel background classes with AMS features yields better performance in 84% of investigated situations, corresponding to an SNR benefit of about 10 dB compared to mel-frequency cepstral coefficient (MFCC) features. (2) On known backgrounds, AMS and MFCCs achieve similar performance, with a small advantage for AMS in negative SNR regimes. (3) Standard voice activity detection (ITU G729.B) performs significantly worse than the classification-based approach.

In this study, a new generic framework for the detection and interpretation of disagreement (“incongruence”) between different classifiers [1] is applied to the problem of detecting novel acoustic objects in an office environment. Using a general model that detects generic acoustic objects (standing out from a stationary background) and specific models tuned to particular sounds expected in the office, a novel object is detected as an incongruence between the models: the general model detects it as a generic object, but the specific models can not identify it as any of the known office-related sources. The detectors are realized using amplitude modulation spectrogram and RASTA-PLP features with support vector machine classification. Data considered are speech and non-speech sounds embedded in real office background at signal-to-noise ratios (SNR) from +20 dB to −20 dB. Our approach yields approximately 90% hit rate for novel events at 20 dB SNR, 75% at 0 dB and reaches chance level below −10 dB.

It is of prime importance in everyday human life to cope with and respond appropriately to events that are not foreseen by prior experience. Machines to a large extent lack the ability to respond appropriately to such inputs. An important class of unexpected events is defined by incongruent combinations of inputs from different modalities and therefore multimodal information provides a crucial cue for the identification of such events, e.g., the sound of a voice is being heard while the person in the field-of-view does not move her lips. In the project DIRAC ("Detection and Identification of Rare Audio-visual Cues") we have been developing algorithmic approaches to the detection of such events, as well as an experimental hardware platform to test it. An audio-visual platform ("AWEAR" - audio-visual wearable device) has been constructed with the goal to help users with disabilities or a high cognitive load to deal with unexpected events. Key hardware components include stereo panoramic vision sensors and 6-channel worn-behind-the-ear (hearing aid) microphone arrays. Data have been recorded to study audio-visual tracking, a/v scene/object classification and a/v detection of incongruencies.

A classification method is presented that detects the presence of speech embedded in a real acoustic background of non-speech sounds. Features used for classification are modulation components extracted by computation of the amplitude modulation spectrogram. Feature selection techniques and support vector classification are employed to identify modulation components most salient for the classification task and therefore considered as highly characteristic for speech. Results show that reliable detection of speech can be performed with less than 10 optimally selected modulation features, the most important ones are located in the modulation frequency range below 10 Hz. Detection of speech in a background of non-speech signals is performed with about 90% test-data accuracy at a signal-to-noise level of 0 dB. Compared to standard ITU G729.B voice activity detection, the proposed method results in increased true positive and reduced false positive rates induced by a real acoustic background.