In severe noise conditions, employing a speech enhancement (SE) model as a front-end serves as a computationally efficient strategy for robust automatic speech recognition (ASR), offering a practical alternative to the costly fine-tuning of largescale ASR systems. However, improvements in human perceptual quality do not necessarily guarantee enhanced machine recognition accuracy, as aggressive noise suppression often introduces distortions and artifacts that obscure fine-grained spectral details and increase word error rates (WERs). To mitigate the discrepancy, we present the Dynamic Distortion Control (DDC) loss, a unified training objective designed to bridge the gap between perceptual fidelity and recognition robustness. Integrated into a time-frequency Transformer architecture, the presented loss addresses the distortion-robustness discrepancy. Experimental results on a LibriSpeech dataset corrupted by noise from DNS Challenge demonstrate the effectiveness of the DDC loss on both perceptual quality and recognition accuracy across diverse noise conditions.

In general, multi-channel source separation has utilized inter-microphone phase differences (IPDs) concatenated with magnitude information in time-frequency domain, or real and imaginary components stacked along the channel axis. However, the spatial information of a sound source is fundamentally contained in the “differences” between microphones, specifically in the correlation between them, while the power of each microphone also provides valuable information about the source spectrum, which is why the magnitude is also included. Therefore, we propose a network that directly leverages a correlation input with phase transform (PHAT)-$\beta$to estimate the separation filter. In addition, the proposed TF-CorrNet processes the features alternately across time and frequency axes as a dual-path strategy in terms of spatial information. Furthermore, we add a spectral module to model source-related direct time-frequency patterns for improved speech separation. Experimental results demonstrate that the proposed TF-CorrNet effectively separates the speech sounds, showing high performance with a low computational cost in the LibriCSS dataset.

In speaker verification, ECAPA-TDNN has shown remarkable improvement by utilizing one-dimensional(1D) Res2Net block and squeeze-and-excitation(SE) module, along with multi-layer feature aggregation (MFA). Meanwhile, in vision tasks, ConvNet structures have been modernized by referring to Transformer, resulting in improved performance. In this paper, we present an improved block design for TDNN in speaker verification. Inspired by recent ConvNet structures, we replace the SE-Res2Net block in ECAPA-TDNN with a novel 1D two-step multi-scale ConvNeXt block, which we call TS-ConvNeXt. The TS-ConvNeXt block is constructed using two separated sub-modules: a temporal multi-scale convolution (MSC) and a frame-wise feed-forward network (FFN). This two-step design allows for flexible capturing of inter-frame and intra-frame contexts. Additionally, we introduce global response normalization (GRN) for the FFN modules to enable more selective feature propagation, similar to the SE module in ECAPA-TDNN. Experimental results demonstrate that NeXt-TDNN, with a modernized backbone block, significantly improved performance in speaker verification tasks while reducing parameter size and inference time. We have released our code1for future studies.

Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed. However, most existing datasets focus on English, developed from pre-existing videos using various prediction models, and have only a small number of multi-view videos. To mitigate the limitations, we constructed the Open Large-scale Korean Audio-Visual Speech (OLKAVS) dataset, which is the largest among publicly available audio-visual speech datasets. The dataset contains 1,150 hours of transcribed audio from 1,107 Korean speakers in a studio setup with nine different viewpoints and various noise situations. We also provide the pre-trained baseline models for two tasks: audiovisual speech recognition and lip reading. We conducted experiments based on the models to verify the effectiveness of multi-modal and multi-view training over uni-modal and frontal-view-only training. We expect the OLKAVS dataset to facilitate multi-modal research in broader areas.

In speech separation, time-domain approaches have successfully replaced the time-frequency domain with latent sequence feature from a learnable encoder. Conventionally, the feature is separated into speaker-specific ones at the final stage of the network. Instead, we propose a more intuitive strategy that separates features earlier by expanding the feature sequence to the number of speakers as an extra dimension. To achieve this, an asymmetric strategy is presented in which the encoder and decoder are partitioned to perform distinct processing in separation tasks. The encoder analyzes features, and the output of the encoder is split into the number of speakers to be separated. The separated sequences are then reconstructed by the weight-shared decoder, which also performs cross-speaker processing. Without relying on speaker information, the weight-shared network in the decoder directly learns to discriminate features using a separation objective. In addition, to improve performance, traditional methods have extended the sequence length, leading to the adoption of dual-path models, which handle the much longer sequence effectively by segmenting it into chunks. To address this, we introduce global and local Transformer blocks that can directly handle long sequences more efficiently without chunking and dual-path processing. The experimental results demonstrated that this asymmetric structure is effective and that the combination of proposed global and local Transformer can sufficiently replace the role of inter- and intra-chunk processing in dual-path structure. Finally, the presented model combining both of these achieved state-of-the-art performance with much less computation in various benchmark datasets.

In this paper, we present a statistical beamforming algorithm as a pre-processing step for robust automatic speech recognition (ASR). By modeling the target speech as a non-stationary Laplacian distribution, a mask-based statistical beamforming algorithm is proposed to exploit both its output and masked input variance for robust estimation of the beamformer. In addition, we also present a method for steering vector estimation (SVE) based on a noise power ratio obtained from the target and noise outputs in independent component analysis (ICA). To update the beamformer in the same ICA framework, we derive ICA with distortionless and null constraints on target speech, which yields beamformed speech at the target output and noises at the other outputs, respectively. The demixing weights for the target output result in a statistical beamformer with the weighted spatial covariance matrix (wSCM) using a weighting function characterized by a source model. To enhance the SVE, the strict null constraints imposed by the Lagrange multiplier methods are relaxed by generalized penalties with weight parameters, while the strict distortionless constraints are maintained. Furthermore, we derive an online algorithm based on an optimization technique of recursive least squares (RLS) for practical applications. Experimental results on various environments using CHiME-4 and LibriCSS datasets demonstrate the effectiveness of the presented algorithm compared to conventional beamforming and blind source extraction (BSE) based on ICA on both batch and online processing.

Beamforming has been one of the most successful approaches using multi-microphones for robust speech recognition. Although a beamforming method, called the “maximum-likelihood distortionless response (MLDR)” beamformer, was recently presented to achieve promising performance, it requires an accurate steering vector for a target speaker in advance like many kinds of beamformers. In this paper, we present a method for steering vector estimation (SVE) by replacing the noise spatial covariance matrix estimate with a normalized version of the variance-weighted spatial covariance matrix estimate for the observed noisy speech signal obtained by the iterative update rule in the MLDR beamforming framework. In addition, an MLDR beamforming method without a steering vector for a target speaker given in advance is presented where the SVE and the beamforming are alternately repeated. Furthermore, an online algorithm based on recursive least squares (RLS) is derived to cope with various practical applications including time-varying situations, and the power method is introduced for further efficient online processing. We also present batch and online convolutional MLDR beamforming with SVE for simultaneous beamforming and dereverberation where the weighted prediction error (WPE) dereverberation and the MLDR beamforming with the SVE were jointly optimized based on the maximum-likelihood estimation (MLE) for a zero-mean complex Gaussian signal with time-varying variances. Moreover, input signals masked by a neural network (NN) for estimating target speech or noise components can be used to further improve the presented beamformers. Experimental results on the CHiME-4 and REVERB challenge datasets demonstrate the effectiveness of the presented methods.

In this letter, we propose a new approach to tracking a target that maneuvers based on the multiple-constant-turns model. Usually, the interactive-multiple-model (IMM) algorithm based on the extended Kalman filter (IMM-EKF) is employed for this problem with successful tracking performance. Recently proposed IMM-particle filtering (IMM-PF) showed outperforming results over IMM-EKF for this nonlinear problem. The proposed approach in this letter is a new framework of PF that adopts the minimax strategy to IMM-PF. The minimax strategy results in the decreased variance of the weights of particles that provides the robustness against the degeneracy phenomenon (a common problem of generic PF). In this letter, we show outperforming results by IMM-minimax-PF over IMM-PF besides the IMM-EKF in terms of estimation accuracy and computational complexity.

This letter addresses acoustic beamforming for robust speech recognition. Beamforming has been demonstrated to be one of the most effective approaches for robust recognition of distant speech using multi-microphones. In this letter, we derive a beamforming method, which we refer to as the “maximum-likelihood distortionless response (MLDR)” beamformer, based on the maximum-likelihood estimation (MLE) of a linear filter, with a distortionless constraint on the steering direction, assuming that the optimal beamformer outputs in the time-frequency domain follow a zero-mean complex Gaussian distribution with time-varying variances. By optimizing the beamformer output variances as well as the filter alternately with iterative update rules, and also by using the moving average of output powers at adjacent frames for robust estimation of an output variance, the MLDR beamformer may minimize the power of a relatively accurate noise component at the output, which resulted in better recognition performance than conventional beamformers. In addition, it can be further improved by initializing the output variances to averaged powers of a neural-network(NN)-masked input signal to estimate target speech powers, which achieved even better performance than compared beamformers exploiting trained NNs.

This letter describes a preprocessing method called subband-based stationary-component suppression method using harmonics and power ratio (SHARP) processing for reverberant speech recognition. SHARP processing extends a previous algorithm called Suppression of Slowly varying components and the Falling edge (SSF), which suppresses the steady-state portions of subband spectral envelopes. The SSF algorithm tends to over-subtract these envelopes in highly reverberant environments when there are high levels of power in previous analysis frames. The proposed SHARP method prevents excessive suppression both by boosting the floor value using the harmonics in voiced speech segments and by inhibiting the subtraction for unvoiced speech by detecting frames in which power is concentrated in high-frequency channels. These modifications enable the SHARP algorithm to improve recognition accuracy by further reducing the mismatch between power contours of clean and reverberated speech. Experimental results indicate that the SHARP method provides better recognition accuracy in highly reverberant environments compared to the SSF algorithm. It is also shown that the performance of the SHARP method can be further improved by combining it with feature-space maximum likelihood linear regression (fMLLR).

The performance of automatic speech recognition (ASR) system is often degraded in adverse real-world environments. In recent times, deep learning has successfully emerged as a breakthrough for acoustic modeling in ASR; accordingly, deep neural network (DNN)-based speech feature enhancement (FE) approaches have attracted much attention owing to their powerful modeling capabilities. However, DNN-based approaches are unable to achieve remarkable performance improvements for speech with severe distortion in the test environments different from training environments. In this letter, we propose a DNN-based FE method where the DNN inputs include preenhanced spectral features computed from multichannel input signals to reconstruct noise-robust features. The preenhanced spectral features are obtained by direction-of-arrival (DOA)-constrained independent component analysis (DCICA) followed by Bayesian FE using a hidden-Markov-model prior, to exploit the capabilities of efficient online target speech extraction and efficient FE with prior information for robust ASR. In addition, noise spectral features computed from DCICA are included for further improvement. Therefore, the DNN is trained to reconstruct a clean spectral feature vector, from a sequence of corrupted input feature vectors in addition to the corresponding preenhanced and noise feature vectors. Experimental results demonstrate that the proposed method significantly improves recognition performance, even in mismatched noise conditions.

We propose a speech dereverberation method in the time-frequency domain, based on an image deblurring algorithm. A reverberant speech magnitude can be modeled as a convolution of a clean speech with a reverberation filter in time-frequency domain. Then, dereverberation problem can be regarded as that of image deblurring. Therefore, the proposed method estimates the clean speech magnitude in the time-frequency domain by using the fast image deconvolution method with priors on sparsity of the clean speech magnitude gradient and exponentially decaying property of reverberation filters along the time axis. Then, scaling the reverberation speech magnitude by a mask obtained from the estimated clean one performs dereverberation. Experimental results show that the described method can enhance speech.

This letter presents an efficient feature enhancement method for reverberant speech recognition that derives a minimum mean square error estimate of clean logarithmic mel-frequency power spectral coefficients (LMPSCs) based on a hidden-Markov-model(HMM) prior. Although an observation model of the reverberant LMPSCs can be simply formulated by coarse modeling of the room impulse response (RIR) , the presented method estimates not only the clean LMPSCs but also the RIR to reflect detailed reverberation. The experimental results indicate that the described method can further reduce relative word error rate (WER) by 18.09% on average compared to a method based on RIR coarse modeling.

This paper presents a robust algorithm to recognize human faces efficiently. Although the principle component analysis (PCA) is one of the most popular feature extraction methods, it requires too much computational load and memory capacity to implement a real-time embedded system for face recognition. To overcome the drawback, we employ the incremental two-directional two-dimensional PCA (I(2D)2PCA) which combines (2D)2PCA to demand much less computational complexity than the conventional PCA and the incremental PCA (IPCA) to adapt the eigenspace only using a new incoming sample datum without memorizing all of the previous trained data. In addition, robustness to illumination variations is addressed by introducing the modified census transform (MCT) which is a local normalization method useful for real-world application and implementation in an embedded system. Experimental results on the Yale Face Database B demonstrate that the proposed method based on the I(2D)2PCA with MCT preprocessing provided efficient and robust face recognition.

This letter presents a multiple sound localization method for non-stationary speech sources in reverberant environments. Although zero-crossing-based interaural time differences (ITDs) were robust to diffuse noise and energy-based onset detection tackled reverberant speech , the onset detection method was sensitive to parameters. The proposed method employs echo-free onset detection , and this letter adds reverberation time estimation to obtain accurate echoes and signal-to-noise ratio estimation to assist the onset detection based on typical properties of acoustic reverberation in selecting ITDs corresponding to source locations. Experimental results indicate the effectiveness of the proposed method in reverberant environments.

In this letter, we describe a method to estimate the time delay for speaker localization in reverberant environments. Based on an adaptive eigenvalue decomposition (AED) algorithm, the method takes the reverberation fully into account by estimating channel impulse responses from a speaker to sensors directly, but it may suffer from whitening effects for temporally correlated natural sounds. Imposing sparse priors on the responses can reduce the temporal whitening and provide a more accurate and robust time delay of speaker location. Experiments demonstrate that the proposed method can efficiently estimate the time delay for speaker localization in reverberant environments.

Undisturbed interaction is essential to provide immersive AR environments. There have been a lot of approaches to interact with VEs (virtual environments) so far, especially in hand metaphor. When the user’s hands are being used for hand-based work such as maintenance and repair, necessity of alternative interaction technique has arisen. In recent research, hands-free gaze information is adopted to AR to perform original actions in concurrence with interaction. [3, 4]. There has been little progress on that research, still at a pilot study in a laboratory setting. In this paper, we introduce such a simple WARS(wearable augmented reality system) equipped with an HMD, scene camera, eye tracker. We propose ‘Aging’ technique improving traditional dwell-time selection, demonstrate AR gallery — dynamic exhibition space with wearable system.

This paper describes an algorithm that efficiently segregates desired speech features from spatially-separated interfering sources in reverberant environments. Although most binaural segregation techniques successfully remove interference components in the absence of reverberation, source segregation in reverberant environments remains a challenging problem. In order to reduce the effects of reverberation, we present a method that dereverberates input signals before they are segregated. The dereverberation filter is estimated from the autocorrelation of the observations and primarily deals with early reflections, while late reflections are effectively suppressed by an inhibitory mechanism that estimates their relative contribution in each time-frequency segment. Information about the salience of the target in a given time-frequency segment based on source separation is combined with the corresponding information based on reverberation suppression through the use of a continually-variable weighting function or mask. Use of the novel reverberation processing results in a relative decrease in WER of 11.5% to 20.9% and use of the combined approaches reduces relative WER by as much as 65.3%.

A modification is proposed to the independent component analysis (ICA)-based filterbank approach in consideration to its structural similarity with binaural auditory model of sound source localization. The estimated sound locations provide an additional cue to the learning algorithm, which is utilized for initialization and imposition of directional constraints on the subband separation networks. Directionally constrained filterbank ICA (DC-FBICA) gives faster convergence and improves separation performance for noisy mixtures having significant spectral overlap among the convolved mixture and the corrupting noise. However, only slight improvement in separation performance is observed when the additive noise is a low frequency noise, although faster convergence is still observed.

This paper describes an algorithm that achieves noise robustness in speech recognition by reconstructing the desired signal from a mixture of two signals using continuously-variable masks. In contrast to current methods which use binary masks, this approach estimates the relative contribution of the desired source in a mixture of sources and reconstructs the desired signal in proportion to its estimated contribution to each time-frequency segment. Estimation of the continuously-variable masks is based on the relationship between the relative intensity of each source and the interaural time difference (ITD). Estimation of the ITD is accomplished using zero-crossing-based methods. It is shown that the use of zero-crossing approaches to estimate ITDs and continuously-variable masks provide better speech recognition accuracy than cross-correlation-based approaches to ITD estimation and binary masks.

We present a new approach to perform independent component analysis (ICA) for convolved mixtures. This approach is based on filter banks, and a simplified network efficiently performs ICA with decimated signals in each subband. Decimation provides much less computational complexity and faster convergence speed than the time domain approach. Furthermore, the approach does not have a performance limitation of the frequency domain approach, and it is able to select the number of filters in the filter bank regardless of reverberation. With an oversampled filter bank, adaptive parameters can be adjusted without any information of other subbands, and the approach is suitable for parallel processing. We verify the effectiveness of the filter bank approach through simulations on adaptive noise cancelling.

An field programmable gate array (FPGA) implementation of independent component analysis (ICA) algorithm is reported for blind signal separation (BSS) and adaptive noise canceling (ANC) in real time. In order to provide enormous computing power for ICA-based algorithms with multipath reverberation, a special digital processor is designed and implemented in FPGA. The chip design fully utilizes modular concept and several chips may be put together for complex applications with a large number of noise sources. Experimental results with a fabricated test board are reported for ANC only, BSS only, and simultaneous ANC/BSS, which demonstrates successful speech enhancement in real environments in real time.