Given an input audio signal where multiple speakers talk over each other, the goal of speech separation is to recover the original signals of each speaker. In this paper we propose a novel sequence modelling method called relative context based on differencing and use it for a speech separation architecture called RCSep. The main advantages of relative context is that it does not require trainable parameters, is very lightweight and highly parallelized. The RCSep model which heavily uses relative context is an extremely efficient source separation model. It has less than 500k trainable parameters, lower memory usage and is significantly faster than all previous source separation methods while still maintaining reasonably high separation accuracy.

This paper proposes a novel method for sequence modelling which we call Seq2Space. The basic idea is to project sequential information into the channel dimension. The Seq2Space layer outperforms the Transformer on every dataset contained in the Long Range Arena (LRA) benchmark as well as on the WSJ0-2 Mix benchmark for single-channel speech separation. Compared to previous methods which were tested on the LRA, the proposed Seq2Space layer does not quite reach the accuracy of the convolution-based methods. It is, however, more than twice as fast as the next fastest method as well as the most memory efficient, and still reaches an average accuracy of 71.15%. On the WSJ0-2Mix, the Seq2Space layer outperforms all other sequence modelling methods in our experiments except for the MEGA layer. By replacing Transformers with the Seq2Space layer on a current SOTA method, we are able to reach 22.8 dB SI-SDR improvement, which is comparable to current SOTA while being significantly faster and more memory efficient during both training and inference.

The problem of single-channel audio source separation is to recover (separate) multiple audio sources that are mixed in a single-channel audio signal (e.g. people talking over each other). Some of the best performing single-channel source separation methods utilize downsampling to either make the separation process faster or make the neural networks bigger and increase accuracy. The problem concerning downsampling is that it usually results in information loss. In this paper, we tackle this problem by introducing SFSRNet which contains a super-resolution (SR) network. The SR network is trained to reconstruct the missing information in the upper frequencies of the audio signal by operating on the spectrograms of the output audio source estimations and the input audio mixture. Any separation method where the length of the sequence is a bottleneck in speed and memory can be made faster or more accurate by using the SR network. Based on the WSJ0-2mix benchmark where estimations of the audio signal of two speakers need to be extracted from the mixture, in our experiments our proposed SFSRNet reaches a scale-invariant signal-to-noise-ratio improvement (SI-SNRi) of 24.0 dB outperforming the state-of-the-art solution SepFormer which reaches an SI-SNRi of 22.3 dB.