Abstract Parameter-efficient fine-tuning (PEFT) for personalizing automatic speech recognition (ASR) has recently shown promise for adapting general population models to atypical speech. However, these approaches assume a priori knowledge of the atypical speech disorder being adapted for—the diagnosis of which requires expert knowledge that is not always available. Even given this knowledge, data scarcity and high inter-/intra-speaker variability further limit the effectiveness of traditional fine-tuning. To circumvent these challenges, we first identify the minimal set of model parameters required for ASR adaptation. Our analysis of each individual parameter’s effect on adaptation performance allows us to reduce Word Error Rate (WER) by half while adapting 0.03% of all weights. Alleviating the need for cohort-specific models, we next propose the novel use of a meta-learned hypernetwork to generate highly individualized, utterance-level adaptations on-the-fly for a diverse set of atypical speech characteristics. Evaluating adaptation at the global, cohort, and individual-level, we show that hypernetworks generalize better to out-of-distribution speakers, while maintaining an overall relative WER reduction of 75.2% using 0.1% of the full parameter budget.

Consumer speech recognition systems do not work as well for many people with speech differences, such as stuttering, relative to the rest of the general population. However, what is not clear is the degree to which these systems do not work, how they can be improved, or how much people want to use them. In this paper, we first address these questions using results from a 61-person survey from people who stutter and find participants want to use speech recognition but are frequently cut off, misunderstood, or speech predictions do not represent intent. In a second study, where 91 people who stutter recorded voice assistant commands and dictation, we quantify how dysfluencies impede performance in a consumer-grade speech recognition system. Through three technical investigations, we demonstrate how many common errors can be prevented, resulting in a system that cuts utterances off 79.1% less often and improves word error rate from 25.4% to 9.9%.

The use of a nearby external microphone for addressing front-back ambiguity in single-microphone hearing aid devices is investigated. Strategic placement of the external microphone is able to provide benefits from the body shielding back-directional noise and therefore information for discriminating between the frontal and back hemispheres. The scattering effects of the body are first analyzed to yield a placement strategy of the external microphone for maximizing the shielding effect of the body against back-directional noise sources while also optimizing for speech intelligibility. Assuming optimal placement, a frontal target source presence probability (FTSPP) estimator is derived. Using the FTSPP estimator, a more comprehensive noise estimator is proposed, which considers both stationary and nonstationary interferers. The performance of the proposed noise estimator is evaluated in its application for postfiltering the output of a binaural beamformer. The effect of postfiltering using the proposed noise estimator is to provide further reduction of directional noise from the lateral and back direction while preserving the frontal target signal. The resulting enhancement with the proposed noise estimator provides a better signal-to-noise ratio and improved objective speech quality and intelligibility of a frontal target speaker compared to the state of the art.

This paper presents a strategy for using an external microphone for enhancing noisy speech in single-microphone completely-in-canal (CIC) hearing aids. The external microphone is placed such that it benefits from the body shielding noise from the back hemisphere. The presented algorithm first enhances the external microphone signal without assuming an exact known location of the external microphone. The proposed algorithm then automatically incorporates the enhanced external microphone signal for post-processing enhancement of a conventional dual-channel binaural beamformer whenever the external microphone has a significant SNR advantage. The overall enhancement scheme avoids error-prone estimations of target voice activity detection and relative transfer functions between the microphones. Unlike single-channel post-processing filters which are limited to reducing stationary or diffuse noise, the proposed postprocessing filter is able to reduce highly non-stationary directional noise from the back hemisphere. The resulting system provides enhancement even for noise arriving from the backward direction, which conventionally is difficult for CIC hearing aids where there exists a front-back ambiguity.