This paper proposes a novel, resource-efficient approach to Visual Speech Recognition (VSR) leveraging speech representations produced by any trained Automatic Speech Recognition (ASR) model. Moving away from the resource-intensive trends prevalent in recent literature, our method distills knowledge from a trained Conformer-based ASR model, achieving competitive performance on standard VSR benchmarks with significantly less resource utilization. Using unlabeled audio-visual data only, our baseline model achieves a word error rate (WER) of 47.4% and 54.7% on the LRS2 and LRS3 test benchmarks, respectively. After fine-tuning the model with limited labeled data, the word error rate reduces to 35% (LRS2) and 45.7% (LRS3). Our model can be trained on a single consumer-grade GPU within a few days and is capable of performing real-time end-to-end VSR on dated hardware, suggesting a path towards more accessible and resource-efficient VSR methodologies.

This work addresses the problem of 3D-localizing and enhancing the speech of one main speaker in noisy multi-speaker hospital environments using a multi-channel microphone array. In our model, we propose conducting speaker localization using a machine learning model based on convolutional recurrent neural networks (CRNN) followed by minimum variance distortionless response (MVDR) beamforming. In addition, to ensure that our speech enhancement module is adaptive when deployed in different environments, we trained a meta learning model. Firstly, in the localization step, an estimation of the direction of arrival (DOA) in the elevation and azimuth planes is executed. This is conducted in a 3D space with the presence of noise, reverberation, and up to two more speakers. Using estimated DOA, the MVDR beamformer then enhances the speech of the main speaker. In order to test our model, we adopted and simulated a real-world problem where the objective was to enhance the speech of a clinician in a noisy intensive care unit (ICU) with the presence of other speakers. Furthermore, in order to validate our model, we adopted a speech-to-text module to evaluate the word error rate. Moreover, we implemented our algorithm on hardware using commercially available components and tested it in a real environment. Results showed that our model outperforms other machine learning and non-machine learning algorithms. Finally, we used our trained Meta Learning model to show that our model can adapt to new environments while maintaining high performance after retraining with only a few-shot recordings.