Interactive real-time streaming applications such as audio-video conferencing, online gaming and app streaming, place stringent requirements on the network in terms of delay, jitter, and packet loss. Many of these applications inherently involve client-to-client communication, which is particularly challenging since the performance requirements need to be met while traversing the public wide-area network (WAN). This is different from the typical situation of cloud-to-client communication, where the WAN can often be bypassed by moving a communication end-point to a cloud “edge”, close to the client. Can we nevertheless take advantage of cloud resources to improve the performance of real-time client-to-client streaming over the WAN?

In this work, we develop a new method for quantization in multistage audio coding. Given a (perceptual) distortion measure and a bit-rate constraint, we analytically derive the optimal rate distribution between subcoders (stages) and the corresponding optimal quantizers using high-rate theory. The analytical solutions for optimal quantizers allow a coder to easily adapt to changes in bit-rate requirements. As an illustration of the new method, we consider quantization in a two-stage sinusoidal/waveform coder that is a widely used combination in audio coding. We show that at low total rates most of the rate should be assigned to the sinusoidal (model-based, subspace) subcoder, while at high total rates most of the rate should be assigned to the waveform (full-space) subcoder. We compare the new method to a reference quantization method that does not use rate-distortion optimization. A significantly higher performance of the new method is shown by means of a listening test.

In this work, we develop a new method for jointly optimal quantization of sinusoidal frequencies, amplitudes, and phases and apply the method to sinusoidal audio coding. This is an extension of an earlier work on quantization of sinusoidal amplitudes and phases to frequencies. The optimization is performed for a set of sinusoids that models a short segment of an audio signal. For a given bit-rate constraint, the optimal quantizers minimize a single-letter weighted distortion measure that accounts for perceptual importance of sinusoids. The quantizers are derived analytically using high-rate theory. The method yields high performance and has a number of practical advantages over conventional sinusoidal quantization methods.

In this work, we present a new method for quantization of sinusoidal amplitudes and phases, and apply the method to sinusoidal coding of speech and audio signals. The method is based on unrestricted polar quantization, where phase quantization accuracy depends on amplitude. Amplitude and phase quantizers are derived under an entropy (average rate) constraint using high-rate assumptions. First, we derive optimal quantizers for one sinusoid and a mean-squared error distortion measure. We provide a detailed analysis of entropy-constrained unrestricted polar quantization, showing its high performance and practicality even at low rates. Second, we find optimal quantizers for a set of sinusoids that model a short segment of an audio signal. The optimization is performed using a weighted error measure that can account for the masking effect in the human auditory system. We find the optimal rate distribution between sinusoids, as well as the corresponding optimal amplitude and phase quantizers, based on the perceptual importance of sinusoids defined by masking. The new method is used in an audio-coding application and is shown to significantly outperform a conventional sinusoidal quantization method where phase quantization accuracy is identical for all sinusoids.

We introduce multivariate block polar quantization (MBPQ). MBPQ minimizes the weighted squared-error distortion for a set of complex variables representing one block of a signal under a resolution constraint for the entire block. MBPQ performs below the lower bound for classical bivariate quantization, both for Gaussian complex variables and for complex variables found from sinusoidal analysis of audio data. Still, it is of similar complexity as traditional polar quantizers. In the case of audio data, we found a performance gain of about 2.5 dB over the best performing conventional resolution-constrained polar quantization (an extension of unrestricted polar quantization).

We introduce multi-variate block polar quantization (MBPQ). MBPQ minimizes a weighted distortion for a set of complex variables representing one block of a signal under a resolution constraint for the entire block. MBPQ allows for different probability distributions in different dimensions of the set of complex variables. It outperforms a block polar quantizer introduced earlier (Pobloth, H. et al., Proc. Eurospeech, p.1097-100, 2003), and unrestricted polar quantization (UPQ), for both Gaussian complex variables and sinusoids found from audio data. In the case of audio data, we found a performance gain of about 2.5 dB over the best performing conventional resolution-constrained polar quantization, UPQ.

We develop a new method for quantization in multistage audio coding. We consider the case of a two-stage sinusoidal/waveform coder. Given a distortion measure and a bit-rate constraint, we analytically derive the optimal rate distribution between subcoders (stages) and the corresponding optimal quantizers, which allows the coder to adapt easily to changes in bit-rate requirements. We verify that the performance, both in terms of signal-to-noise ratio (SNR) and perceptual quality, is higher if the input to the second stage is obtained by subtracting the quantized first-stage reconstruction from the original signal, as opposed to subtracting the unquantized reconstruction.

In this work, we develop entropy-constrained unrestricted polar quantizers, where phase quantization depends on the input amplitude. Formulas for amplitude and phase quantization point densities are derived under high-rate assumptions. It is shown that the mean-squared distortion is decreased considerably as compared to strictly polar quantization and approaches that of scalar rectangular quantization asymptotically with increasing rate. The unrestricted polar quantization is generalized to include a weighted error measure, such that it accounts for masking effects of the human auditory system. Both amplitude and phase quantization depend on the perceptual importance of sinusoids. The new method is applied to a sinusoidal audio coder, and is shown to outperform a conventional sinusoidal quantization method where the number of phase quantization bits is the same for all audible sinusoids.

We propose a segment-based matching-pursuit algorithm where the psychoacoustical properties of the human auditory system are taken into account. Rather than scaling the dictionary elements according to auditory perception, we define a psychoacoustic-adaptive norm on the signal space that can be used for assigning the dictionary elements to the individual segments in a rate-distortion optimal way. The new algorithm is asymptotically equal to signal-to-mask-ratio-based algorithms in the limit of infinite-analysis window length. However, the new algorithm provides a significantly improved selection of the dictionary elements for finite window length.

We propose a segment-based matching pursuit algorithm where the psychoacoustical properties of the human auditory system are taken into account. Rather than scaling the dictionary elements according to auditory perception, we define a psychoacoustic-adaptive norm on the signal space which can be used for assigning the dictionary elements to the individual segments in a rate-distortion optimal manner. The new algorithm is asymptotically equal to signal-to-mask ratio based algorithms in the limit of infinite analysis window length. However, the new algorithm provides a significantly improved selection of the dictionary elements for finite window length.

We propose a method for efficient representation of transients in audio signals. We estimate the transient component of an original audio signal and modify the locations of the transients in such a way that the transients can occur only at locations defined by a relatively coarse time grid. This procedure allows an efficient representation of transients with damped sinusoids. We also verify that the introduced modifications do not result in a perceptual difference between the original and the modified audio signals.

In this paper, we elaborate on the issue of analysis-synthesis consistency in sinusoidal coding. Our analysis is based on windowed sinusoids, and uses the same amplitude-complementary window as is used in the overlap-add synthesis. Reconstructions of the neighboring segments are taken into account when forming a particular analysis segment. Sinusoidal estimation is based on a perceptual criterion. In our new procedure, when analyzing the current segment we take advantage of the forward masking effect due to estimated sinusoids in the previous segments (possibly overlapping with the current segment). Experimental results verify that the number of sinusoids can be reduced significantly with our time masking model, without introducing perceptual artifacts in the reconstructed signal.