Text-to-audio (TTA), which generates audio signals from textual descriptions, has received huge attention in recent years. However, recent works focused on text to monaural audio only. As we know, spatial audio provides more immersive auditory experience than monaural audio, e.g. in virtual reality. To address this issue, we propose a text-to-spatial-audio (TTSA) generation framework named DualSpec. Specifically, it first trains variational autoencoders (VAEs) for extracting the latent acoustic representations from sound event audio. Then, given text that describes sound events and event directions, the proposed method uses the encoder of a pretrained large language model to transform the text into text features. Finally, it trains a diffusion model from the latent acoustic representations and text features for the spatial audio generation. In the inference stage, only the text description is needed to generate spatial audio. Particularly, to improve the synthesis quality and azimuth accuracy of the spatial sound events simultaneously, we propose to use two kinds of acoustic features. One is the Mel spectrograms which is good for improving the synthesis quality, and the other is the short-time Fourier transform spectrograms which is good at improving the azimuth accuracy. We provide a pipeline of constructing spatial audio dataset with text prompts, for the training of the VAEs and diffusion model. We also introduce new spatial-aware evaluation metrics to quantify the azimuth errors of the generated spatial audio recordings. Experimental results demonstrate that the proposed method can generate spatial audio with high directional and event consistency.

Target speaker extraction (TSE) aims to isolate a target speaker’s voice from mixed speech using additional cues. Most TSE models rely on air-conduction (AC) signals, which are easily affected by interfering speakers and background noise. In contrast, bone-conduction (BC) speech is naturally resistant to ambient noise and captures only the target speaker’s voice. Existing BC-AC fusion methods have only been applied in conventional speech enhancement, lacking exploration in TSE applications. To leverage the advantages of BC signals, we propose the Multi-modal Bone-conduction Enhancement Network (MBENet), the first BC-AC fusion model designed for the TSE task. To further enhance performance, we introduce multi-task learning by incorporating the bandwidth extension of the BC channel as an auxiliary task. Experimental results show that our casual model outperforms existing approaches in challenging environments.

Speech separation using ad-hoc microphone arrays has been explored, but there is still significant room for improvement, especially in complex scenarios with varying channel conditions. Co-attention, a feature fusion mechanism, is widely used in multimodal fusion to capture the cooperation between modalities and enhance the representation of extracted features. In this paper, we propose a co-attention-based multi-channel model for speech separation with ad-hoc microphone arrays. The co-attention mechanism is integrated into the model to enhance the interaction between different speakers across multiple channels, enabling efficient channel fusion. To the best of our knowledge, this is the first work to apply co-attention for speech separation. Experimental results demonstrate that the proposed method significantly outperforms existing approaches, underscoring the importance of co-attention in optimizing channel fusion for speech separation in challenging acoustic environments.

Multi-dimensional speaker localization (SL) aims to estimate the two- or three-dimensional locations of speakers. A recent advancement in multi-dimensional SL is the end-to-end deep neural networks (DNNs) with ad-hoc microphone arrays. This method transforms the SL problem into a classification problem, i.e. a problem of identifying the grids where speakers are located. However, the classification formulation has two closely connected weaknesses. Firstly, this approach introduces quantization error, which needs a large number of grids to mitigate the error. However, increasing the number of grids leads to the curse of dimensionality. To address the problems, we propose an efficient multi-dimensional SL algorithm, which has the following three novel contributions. First, we decouple the high-dimensional grid partitioning intoaxis partitioning, which substantially mitigates the curse-of-dimensionality. Particularly, for the multi-speaker localization problem, we employ a separator to circumvent the permutation ambiguity of the axis partitioning in the inference stage. Second, we introduce a comprehensiveunbiased label distributionscheme to further eliminate quantization errors. Finally, a set of data augmentation techniques are proposed, including coordinate transformation, stochastic node selection, and mixed training, to alleviate overfitting and sample imbalance problems. The proposed methods were evaluated on both simulated and real-world data, and the experimental results confirm the effectiveness.

Recently, an end-to-end two-dimensional sound source localization algorithm with ad-hoc microphone arrays formulates the sound source localization problem as a classification problem. The algorithm divides the target indoor space into a set of local areas, and predicts the local area where the speaker locates. However, the local areas are encoded by one-hot code, which may lose the connections between the local areas due to quantization errors. In this paper, we propose a new soft label coding method, named label smoothing, for the classification-based two-dimensional sound source location with ad-hoc microphone arrays. The core idea is to take the geometric connection between the classes into the label coding process. The first one is named static soft label coding (SSLC), which modifies the one-hot codes into soft codes based on the distances between the local areas. Because SSLC is handcrafted which may not be optimal, the second one, named dynamic soft label coding (DSLC), further rectifies SSLC, by learning the soft codes according to the statistics of the predictions produced by the classification-based localization model in the training stage. Experimental results show that the proposed methods can effectively improve the localization accuracy.