In the history of audio and acoustic signal processing, perceptual audio coding has certainly excelled as a bright success story by its ubiquitous deployment in virtually all digital media devices, such as computers, tablets, mobile phones, set-top-boxes, and digital radios. From a technology perspective, perceptual audio coding has undergone tremendous development from the first very basic perceptually driven coders (including the popular mp3 format) to today’s full-blown integrated coding/rendering systems. This paper provides a historical overview of this research journey by pinpointing the pivotal development steps in the evolution of perceptual audio coding. Finally, it provides thoughts about future directions in this area.

The term “immersive audio” is frequently used to describe an audio experience that provides the listener the sensation of being fully immersed or “present” in a sound scene. This can be achieved via different presentation modes, such as surround sound (several loudspeakers horizontally arranged around the listener), 3D audio (with loudspeakers at, above, and below listener ear level), and binaural audio to headphones. This article provides an overview of two recent standards that support the bitrate-efficient carriage of high-quality immersive sound. The first is MPEG-H 3D audio, which is a versatile standard that supports multiple immersive sound signal formats (channels, objects, and higher order ambisonics) and is now being adopted in broadcast and streaming applications. The second is MPEG-I immersive audio, an extension of 3D audio, currently under development, which is targeted for virtual and augmented reality applications. This will support rendering of fully user-interactive immersive sound for three degrees of user movement [three degrees of freedom (3DoF)], i.e., yaw, pitch, and roll head movement, and for six degrees of user movement [six degrees of freedom (6DoF)], i.e., 3DoF plus translational x, y, and z user position movements.

An algorithm for covert digital audio watermarking is presented. The method embeds a low-rate (in the neighborhood of 20–30 b/sec) watermark via perceptually insignificant long term multiband phase modulation, The watermark is very difficult to either discover or recover without the original (unmodulated) signal. The algorithm is shown to be robust to perceptual coding.

MPEG-7 is a new ISO standard that facilitates searching for media content much as current text-based search engines ease retrieval of HTML content. This paper gives an overview of the MPEG-7 audio standard, in terms of the applications it might support, its structure, the process by which it was developed, and its specific descriptors and description schemes.

A watermark is a data stream inserted into multimedia content. It contains information relevant to the ownership or authorized use of the content. A watermark which could be recovered without a priori knowledge of the identity of the content could be used by Web search mechanisms to flag unauthorized distribution of the content. Since media will be compressed on these sites, a mark detection algorithm that operated in the compressed domain would be useful. We describe a watermark algorithm which operates in the compressed domain and does not require a reference.

MPEG-4 standardizes natural audio coding at bit rates ranging from 2 kbit/s, suitable for intelligible speech coding, to 64 kbit/s per channel, suitable for high-quality audio coding. Within this range, three categories of coding are defined: parametric coding, code excited linear predictive coding (CELP) and time/frequency (T/F) coding. The unique contribution of MPEG-4 audio is that not only does it scale across a wide range of bit rates, but it also scales across a broad set of other parameters, such as sampling rate, bandwidth, voice pitch and complexity. This paper presents an overview of the MPEG-4 natural audio coding framework and each of its component coding techniques.

The author describes a voice codec with a 20 to 7000 Hz bandwidth that compresses to 2 bits per sample at a sample rate of 16 kHz. The algorithm is based on a transform coder that employs dynamic bit allocation in the frequency domain using a perceptual noise weighting criterion, and by Huffman compression of the quantized transform signal. It is well suited to ISDN (integrated services digital network) channels since it permits two coded voice signals to be carried on a single 64 kbps ISDN B channel. A two-channel coder has been implemented in real-time using three AT&T DSP32C signal processors and runs on the public switched ISDN network. The one-way communication delay of this hardware is 75 msec.>

Database client hardware and associated algorithms that can be used to call a remote database server using commercial ISDN lines for retrieval of image information are described. Compressed images are stored in the server and transferred over the ISDN channel at 64 kb/s. This information is reconstructed at the client using a single 50-MHz AT&T WE DSP32C signal processor and auxiliary display hardware. Compression is accomplished using a 4 by 4 subband filterbank and a perceptual error measure operating in the subband domain. The perceptual measure determines what information can be discarded, with the rest quantized using differential pulse code modulation (DPCM), Huffman encoded and stored at the server. When retrieved, the client performs the inverse operations. A color image of 512 by 480 pixels can be compressed to 0.7 b/pixel, transferred to the client in 3 s, and reconstructed in 6 s, with results nearly indistinguishable from the original.>

A real-time 16 kb/s waveform coder which uses a combination of subband analysis/synthesis and vector quantization (VQ) to achieve a one-way communication delay of 20 ms is described. Gain-shape decomposition is used to vector-quantize the outputs of a generalized quadrature mirror filter bank. The short-time filter bank energy (gain) is used to dynamically allocate bits for coding the gain-normalized filter outputs (shape). The coder-decoder, implemented on a single AT&T DSP32 signal processing chip with 256 kB of off-chip memory for codebook storage, has a typical segmental SNR of 20.44 dB for sentences outside the training set.>

Two very different subband coders are described. The first is a modified dynamic bit-allocation-subband coder (D-SBC) designed for variable rate coding situations and easily adaptable to noisy channel environments. It can operate at rates as low as 12 kb/s and still give good quality speech. The second coder is a 16-kb/s waveform coder, based on a combination of subband coding and vector quantization (VQ-SBC). The key feature of this coder is its short coding delay, which makes it suitable for real-time communication networks. The speech quality of both coders has been enhanced by adaptive postfiltering. The coders have been implemented on a single AT&T DSP32 signal processor.>

Most objective speech quality measures studied in the past have attempted to measure overall acceptability or naturalness. It is intrinsically difficult to design such measures because many perceptually dissimilar distortions may be judged to have equal acceptability. This paper describes objective measures which estimate the perceptually specific (parametric) subjective qualities of the Diagnostic Acceptability Measure. The narrower scope of these subjective qualities promotes a closer coupling between physical quantities and perceptual qualities in a given quality category, resulting in a more accurate objective measures.

This paper explores the utility of designing pointwise nonlinear transforms for simple objective speech quality measures using polynomial regression analysis. A method for the design and evaluation of these nonlinearities is presented and the performance of the resulting measures is discussed. The results of this study indicate that polynomial regression can produce objective measures which are comparable with those obtained using frequency variant or time classified design methodologies.

This paper discusses the results of a large and continuing study of objectively computable speech quality measures. The paper specifically addresses four topics. The first topic is the design of a general testing procedure for estimating the performance of objective speech quality measures. The second topic is a comparative study of 1500 parametric variations of commonly used objective speech quality measures. The third topic is the design of new complex objective speech quality measures based on the data available in the subjective quality data base. The final topic is the development of techniques for the design of improved speech coders based on the minimization of complex speech distortion measures.