Low latency models are critical for real-time speech enhancement applications, such as hearing aids and hearables. However, the sub-millisecond latency space for resource-constrained hearables remains underexplored. We demonstrate speech enhancement using a computationally efficient minimum-phase FIR filter, enabling sample-by-sample processing to achieve mean algorithmic latency of 0.32 ms to 1.25 ms. With a single microphone, we observe a mean SI-SDRi of 4.1 dB. The approach shows generalization with a DNSMOS increase of 0.2 on unseen audio recordings. We use a lightweight LSTM-based model of 626k parameters to generate FIR taps. Using a real hardware implementation on a low-power DSP, our system can run with 376 MIPS and a mean end-to-end latency of 3.35 ms. In addition, we provide a comparison with existing low-latency spectral masking techniques. We hope this work will enable a better understanding of latency and can be used to improve the comfort and usability of hearables.

Wearable vibrotactile devices have many potential applications, including sensory substitution for accessibility and notifications. Currently, vibrotactile experimentation is done using large lab setups. However, most practical applications require standalone on-body devices and integration into small form factors. Such integration is time-consuming and requires expertise.

Today's wearable and mobile devices typically use separate hardware components for sensing and actuation. In this work, we introduce new opportunities for the Linear Resonant Actuator (LRA), which is ubiquitous in such devices due to its capability for providing rich haptic feedback. By leveraging strategies to enable active and passive sensing capabilities with LRAs, we demonstrate their benefits and potential as self-contained I/O devices. Specifically, we use the back-EMF voltage to classify if the LRA is tapped, touched, as well as how much pressure is being applied. The back-EMF sensing is already integrated into many motor and LRA drivers. We developed a passive low-power tap sensing method that uses just 37.7 uA. Furthermore, we developed active touch and pressure sensing, which is low-power, quiet (2 dB), and minimizes vibration. The sensing method works with many types of LRAs. We show applications, such as pressure-sensing side-buttons on a mobile phone. We have also implemented our technique directly on an existing mobile phone's LRA to detect if the phone is handheld or placed on a soft or hard surface. Finally, we show that this method can be used for haptic devices to determine if the LRA makes good contact with the skin. Our approach can add rich sensing capabilities to the ubiquitous LRA actuators without requiring additional sensors or hardware.

We have implemented a near-ultrasonic communication protocol in the 18.5-20 kHz band, which is inaudible to most humans, using commodity smartphone speakers and microphones to transmit and receive signals. The protocol described in this paper is a component of Google's Nearby platform, where near-ultrasound signals are used to establish copresence between nearby devices by transmitting a short token. High-frequency sound does not pass through walls (most energy is reflected), so identified devices are constrained to approximately the same room, “within earshot” of one another. Our protocol has a raw data rate of 94.5 b/s, and we find in real indoor environments that transmission between mobile devices is reliable at 2 m distance and often works at 10 m. We use direct-sequence spread spectrum modulation, which makes it highly robust to multipath, motion, and narrowband noise. We use a 127-chip pseudorandom code, repeating once per data symbol, and modulate its amplitude with orthogonal sine waveforms encoding 4-bit symbol values. We add the orthogonal sines to a constant “pedestal,” which is inefficient in an information-theoretic sense, but makes synchronization easier. We describe a robust and computationally efficient transmitter and receiver implementations and show experiments on real and simulated data.

Robust and far-field speech recognition is critical to enable true hands-free communication. In far-field conditions, signals are attenuated due to distance. To improve robustness to loudness variation, we introduce a novel frontend called per-channel energy normalization (PCEN). The key ingredient of PCEN is the use of an automatic gain control based dynamic compression to replace the widely used static (such as log or root) compression. We evaluate PCEN on the keyword spotting task. On our large rerecorded noisy and far-field eval sets, we show that PCEN significantly improves recognition performance. Furthermore, we model PCEN as neural network layers and optimize high-dimensional PCEN parameters jointly with the keyword spotting acoustic model. The trained PCEN frontend demonstrates significant further improvements without increasing model complexity or inference-time cost.

This paper evaluated psychophysically an ultra-low-power, analog biomimetic cochlear-implant (CI) processor filterbank architecture, which was recently proposed and demonstrated in hardware. The architecture/strategy emulates the lateral inhibition (LI) mechanism by employing an automatic gain control (AGC) scheme that is coupled across One-Zero-Gammatone-Filter (OZGF) channels. The OZGF filtering and the coupled channel AGC were tested respectively in two experiments, where normal-hearing listeners were required to listen to sentences and recognise key words therein. The sentences were mixed with a steady background noise at signal-to-noise ratios (SNRs) of -6, -3 and 0 dB, and processed through a noise-excited envelope vocoder serving as the acoustic simulator of cochlear implants. In the first experiment, the OZGF filtering was compared with cascaded bandpass biquad filtering employed in recent attempts towards fully-implantable CI processing, in terms of the resulting intelligibility. The results showed that the sharply tuned OZGF response did not degrade intelligibility despite the very limited number (16) of channels used. In the second experiment, the sentences were processed in two multi-channel compression systems, one with the channel AGC coupled, whilst another uncoupled. The results showed that the AGC-coupled system was significantly advantageous, and the improvement averaged across the SNRs is 31 percentage points. Furthermore, this compressive system results in no significant decrease in intelligibility when compared to the linear filtering systems investigated in the first experiment. Thus, the coupled channel AGC may be considered as a potential solution to the limited spectral contrast of current CI systems, which may partially account for their noise-susceptibility.

The front end of the human auditory system, the cochlea, converts sound signals from the outside world into neural impulses transmitted along the auditory pathway for further processing. The cochlea senses and separates sound in a nonlinear active fashion, exhibiting remarkable sensitivity and frequency discrimination. Although several electronic models of the cochlea have been proposed and implemented, none of these are able to reproduce all the characteristics of the cochlea, including large dynamic range, large gain and sharp tuning at low sound levels, and low gain and broad tuning at intense sound levels. Here, we implement the `Cascade of Asymmetric Resonators' (CAR) model of the cochlea on an FPGA. CAR represents the basilar membrane filter in the `Cascade of Asymmetric Resonators with Fast-Acting Compression' (CAR-FAC) cochlear model. CAR-FAC is a neuromorphic model of hearing based on a pole-zero filter cascade model of auditory filtering. It uses simple nonlinear extensions of conventional digital filter stages that are well suited to FPGA implementations, so that we are able to implement up to 1224 cochlear sections on Virtex-6 FPGA to process sound data in real time. The FPGA implementation of the electronic cochlea described here may be used as a front-end sound analyser for various machine-hearing applications.

Online video presents a great opportunity for up-and-coming singers and artists to be visible to a worldwide audience. However, the sheer quantity of video makes it difficult to discover promising musicians. We present a novel algorithm to automatically identify talented musicians using machine learning and acoustic analysis on a large set of "home singing" videos. We describe how candidate musician videos are identified and ranked by singing quality. To this end, we present new audio features specifically designed to directly capture singing quality. We evaluate these vis-a-vis a large set of generic audio features and demonstrate that the proposed features have good predictive performance. We also show that this algorithm performs well when videos are normalized for production quality.

A key problem in using the output of an auditory model as the input to a machine-learning system in a machine-hearing application is to find a good feature-extraction layer. For systems such as PAMIR (passive-aggressive model for image retrieval) that work well with a large sparse feature vector, a conversion from auditory images to sparse features is needed. For audio-file ranking and retrieval from text queries, based on stabilized auditory images, we took a multi-scale approach, using vector quantization to choose one sparse feature in each of many overlapping regions of different scales, with the hope that in some regions the features for a sound would be stable even when other interfering sounds were present and affecting other regions. We recently extended our testing of this approach using sound mixtures, and found that the sparse-coded auditory-image features degrade less in interference than vector-quantized MFCC sparse features do. This initial success suggests that our hope of robustness in interference may indeed be realizable, via the general idea of sparse features that are localized in a domain where signal components tend to be localized or stable.

Auditory filter models have a history of over a hundred years, with explicit bio-mimetic inspiration at many stages along the way. From passive analogue electric delay line models, through digital filter models, active analogue VLSI models, and abstract filter shape models, these filters have both represented and driven the state of progress in auditory research. Today, we are able to represent a wide range of linear and nonlinear aspects of the psychophysics and physiology of hearing with a rather simple and elegant set of circuits or computations that have a clear connection to underlying hydrodynamics and with parameters calibrated to human performance data. A key part of the progress in getting to this stage has been the experimental clarification of the nature of cochlear nonlinearities, and the modelling work to map these experimental results into the domain of circuits and systems. No matter how these models are built into machine-hearing systems, their bio-mimetic roots will remain key to their performance. In this paper we review some of these models, explain their advantages and disadvantages and present possible ways of implementing them. As an example, a continuous-time analogue CMOS implementation of the One Zero Gammatone Filter (OZGF) is presented together with its automatic gain control that models its level-dependent nonlinear behaviour.

To create systems that understand the sounds that humans are exposed to in everyday life, we need to represent sounds with features that can discriminate among many different sound classes. Here, we use a sound-ranking framework to quantitatively evaluate such representations in a large-scale task. We have adapted a machine-vision method, the passive-aggressive model for image retrieval (PAMIR), which efficiently learns a linear mapping from a very large sparse feature space to a large query-term space. Using this approach, we compare different auditory front ends and different ways of extracting sparse features from high-dimensional auditory images. We tested auditory models that use an adaptive pole-zero filter cascade (PZFC) auditory filter bank and sparse-code feature extraction from stabilized auditory images with multiple vector quantizers. In addition to auditory image models, we compare a family of more conventional mel-frequency cepstral coefficient (MFCC) front ends. The experimental results show a significant advantage for the auditory models over vector-quantized MFCCs. When thousands of sound files with a query vocabulary of thousands of words were ranked, the best precision at top-1 was 73% and the average precision was 35%, reflecting a 18% improvement over the best competing MFCC front end.

Techniques to recreate sounds from perceptual displays known as cochleagrams and correlograms are developed using a convex projection framework. Prior work on cochlear-model inversion is extended to account for rectification and gain adaptation. A prior technique for phase recovery in spectrogram inversion is combined with the synchronized overlap-and-add technique of speech rate modification, and is applied to inverting the short-time autocorrelation function representation in the auditory correlogram. Improved methods of initial phase estimation are explored. A range of computational cost options, with and without iteration, produce a range of quality levels from fair to near perfect.>

The design and development of an application-specific integrated circuit (ASIC) to implement the filtering and AGC sections of a new digital approximation to the Lyon/Mead analog cochlea model (1988, 1989) are described. The ASIC uses a fully synchronous bit-serial design methodology originally developed for MOS technologies by Denyer and Renshaw (1985) and adaptive subsequently for use with low cost standard cell design tools by Summerfield and Jabri (1989). The resulting device is an example of a new class of signal processing ASIC, referred to as an application-specific signal processor (ASSP). As well as presenting details of the filter and AGC structure of the new cochlea model, the paper describes the bit-serial design methodology used to implement the algorithm in real-time and in low cost ASIC form. The approach is a generic one and can be applied to a wide range of speech signal processing functions.>

A pitch detector based on Licklider's (1979) duplex theory of pitch perception was implemented and tested on a variety of stimuli from human perceptual tests. It is believed that this approach accurately models how people perceive pitch. It is shown that it correctly identifies the pitch of complex harmonic and inharmonic stimuli and that it is robust in the face of noise and phase changes. This perceptual pitch detector combines a cochlear model with a bank of autocorrelators. By performing an independent autocorrelation for each channel, the pitch detector is relatively insensitive to phase changes across channels. The information in the correlogram is filtered, nonlinearly enhanced, and summed across channels. Peaks are identified and a pitch is then proposed that is consistent with the peaks.>

An analog electronic cochlea has been built in CMOS VSLI technology using micropower techniques to achieve the goal of usefulness via realism. The key point of the model and circuits is that a cascade of simple, nearly linear, second-order filter stages with controllable Q parameters suffices to capture the physics of the fluid-dynamic traveling-wave system in the cochlea, including the effects of adaptation and active gain involving the outer hair cells. Measurements on the test chip suggest that the circuit matches both the theory and observations from real cochlea.>

We have conducted speaker-independent isolated digit recognition experiments using vector quantized cochleagrams. Without the use of time order information, we were able to achieve a recognition rate of 97.24%. With a modified Viterbi algorithm we achieved a rate of 98.38%. Since we achieved a 98.05% recognition rate with a scheme that did static pattern matching on the first and second time-halves of our utterances, we must call into question the effectiveness with which the Viterbi algorithm uses time order information. Our results also lead us to conclude that future progress may depend on our ability to construct more sophisticated vector quantizers.

Scale-space filtering, proposed by Witkin (ICASSP 84) for describing natural structure in one-dimensional signals, has been extended for application to segmentation and description of vector-valued functions of time, such as speech spectrograms. By analyzing the rate of change of a vector trajectory at many different scales of time-smoothing, a tree of natural segments can be constructed. At various levels in the tree (i.e., at various scales), these segments are found to agree well with the kind of linguistically and perceptually important segments that spectrogram readers use to describe sound patterns of speech. Scale-space segmentations of cochleagrams (spectrograms based on a computational model of the peripheral auditory system) have been experimentally applied to word recognition. Recognition using fixed-scale segmentations with finite-state word models and a Viterbi search has led to speaker-independent digit recognition accuracies of greater than 97%, about the same as in tests with non-segmented cochleagrams. More complex recognition algorithms that use the segmentation tree are being developed, and scale-space experiments with connected digits and sentences are also underway.

Scale-space filtering represents one method for automatically extracting both coarse and fine-grained units from the speech signal. We examine the acoustic-phonetic structure of segments obtained by scale-space filtering of cochleagrams, and report on the correspondences between scale-space segments which are automatically derived and hand-marked acoustic-phonetic segments. The major advantage of this proposal for segmentation is the flexibility of the data structure.

A computational model of the cochlea has been subjected to a number of experiments to clarify its relation to psychophysical and physiological reality. The experiments are useful in calibrating the model, but more importantly, they are useful in explaining how seemingly contradictory data can be accounted for by simple mechanisms. For example, measurements of threshold tuning curves and masking functions show clearly that threshold tuning curves are sharper than the cochlear mechanical filter transfer functions, without requiring a second tuned mechanism. Similarly, critical bandwidths or spread of masking are seen to be wider than the filter transfer functions. These effects are shown to be consequences of the coupled automatic gain control model which was originally introduced as an aid to handling a large dynamic range; the AGC's effect on tuning is a side effect.

The Multi-Serial Signal Processor (MSSP), a custom bit-serial multiprocessor for signal processing, has been designed and built for application to real-time speech processing. The MSSP is intended to be particularly efficient and well-suited for channel-organized algorithms such as filterbanks, with one or more channels running on each processing element. The particular algorithms that motivated the design of this machine are computational models of hearing, for use in a speech recognizer front-end. An unusual optimizing compiler is used to convert fairly conventional sequential instructions for a channel into efficient microcode that concurrently executes the several stages and many channels of the models.

Explicit neuron firing models are investigated for use in computational modeling of auditory processing. Models for primary auditory neurons are driven by the receptor signal from a hair cell model, which is driven in turn by a filtering model of basilar membrane motion. The output of the primary auditory neuron model is a times-of-firing representation of neural signals. Certain types of processing, such as auto-correlation and cross-correlation, are very simple with this representation, not requiring multiplication. The neuron model used is a leaky-integrate-to-threshold model with a refractory period. Several neurons are modeled at each hair cell, or filter channel. It is found in experiments with these models that the detailed time-of-firing information contains most of the cues of speech formants, pitch, direction, etc. The more conventionally studied firing rate vs. place representation misses important aspects of these cues. Models of pitch perception, binaural directional perception, and sound separation are being based on the cochlear and neural models. The models are all implemented as computational algorithms, and are used in support of related speech recognition and hearing research.

Multiple sound signals, such as speech and interfering noises, can be fairly well separated, localized, and interpreted by human listeners with normal binaural hearing. The computational model presented here, based on earlier cochlear modeling work, is a first step at approaching human levels of performance on the localization and separation tasks. This combination of cochlear and binaural models, implemented as real-time algorithms, could provide the front end for a robust sound interpretation system such as a speech recognizer. The cochlear model used is basically a bandpass filterbank with frequency channels corresponding to places on the basilar membrane; filter outputs are half-wave rectified and amplitude-compressed, maintaining fine time resolution. In the binaural model, outputs of corresponding frequency channels from the two ears are combined by cross-correlation. Peaks in the short-time cross-correlation functions are then interpreted as direction. With appropriate preprocessing, the correlation peaks integrate cues based on signal phase, envelope modulation, onset time, and loudness. Based on peaks in the correlation functions, sources can be recognized, localized, and tracked. Through quickly varying gains, sound fragments are separated into streams representing different sources. Preliminary tests of the algorithms are very encouraging.

We claim that speech analysis algorithms should be based on computational models of human audition, starting at the ears. While much is known about how hearing works, little of this knowledge has been applied in the speech analysis field. We propose models of the inner ear, or cochlea, which are expressed as time- and place-domain signal processing operations; i.e. the models are computational expressions of the important functions of the cochlea. The main parts of the models concern mechanical filtering effects and the mapping of mechanical vibrations into neural representation. Our model cleanly separates these effects into time-invariant linear filtering based on a simple cascade/parallel filterbank network of second-order sections, plus transduction and compression based on half-wave rectification with a nonlinear coupled automatic gain control network. Compared to other speech analysis techniques, this model does a much better job of preserving important detail in both time and frequency, which is important for robust sound analysis. We discuss the ways in which this model differs from more detailed cochlear models.

This paper presents data and criteria to assess and guide the design of modems for coded noncoherent communication systems subject to practical system constraints of powerS, bandwidthW, noise spectral density N0, coherence time Tc, and number of orthogonal signalsM. Three basic receiver types are analyzed for the noncoherent multifrequency-shift keying (MFSK) additive white Gaussian noise channel: hard decision, unquantized (optimum), and quantized (soft decision). Channel capacity and computational cutoff rate Rcompare computed for each type and presented as functions of the predetection signal-to-noise ratioST/N_{0}and the number of orthogonal signalsM = 2TW. This relates the channel constraints of power, bandwidth, coherence time, and noise power to the optimum choice of signal durationT \leq T_{c}and signal numberM.

Communication channels with no transmission at dc require data encoding to eliminate dc from the signal. Many coding schemes have been developed for this purpose. The scheme discussed here uses code blocks with equal numbers of plus and minus pulses, according to a previously published algorithm. This code is compared to a three-level "bipolar" coding scheme in terms of spectral characteristics, efficiency, crosstalk noise, power requirements, and hardware requirements. Circuitry is described for coders, decoders, and repeaters. Extensions to other code types are discussed briefly.