This paper introduces high-quality audio coding using psychoacoustic models. This technology is now abundant, with gadgets named after a standard (mp3 players) and the ability to play high-quality audio from literally billions of devices. The usual paradigm for these systems is based on filterbanks, followed by quantization and coding, controlled by a model of human hearing. The paper describes the basic technology, theoretical framework to apply to check for optimality, and the most prominent standards built on the basic ideas and newer work.

In this panel, each panelist will present their view of the current state-of-the-art of research and product innovations in the three major areas of multimedia experience: visual, auditory and gaming. We will discuss examples of innovation that enhance the consumption and sharing of multimedia (video, audio, graphics etc.) and thus increase quality of user experience. Another major focus of this panel is to open the discussion on how to innovate new multimedia user experiences.

In emerging audio technology applications, there is a need for decompositions of audio signals into oversampled subband components with time-frequency resolution which mimics that of the cochlear filter bank and with high aliasing attenuation in each of the subbands independently, rather than aliasing cancellation properties. We present a design of nearly perfect reconstruction nonuniform oversampled filter banks which implement signal decompositions of this kind.

An algorithm for covert digital audio watermarking is presented. The method embeds a low-rate (in the neighborhood of 20–30 b/sec) watermark via perceptually insignificant long term multiband phase modulation, The watermark is very difficult to either discover or recover without the original (unmodulated) signal. The algorithm is shown to be robust to perceptual coding.

A spatial noise shaping (SNS) method based on human visual sensitivity is presented. The method exploits the capability of frequency domain linear prediction for spatial envelope retrieval. It effectively shapes (or hides) the noise of an image in areas which are not sensitive to human vision so that the resulting image is more pleasant to human eyes. The noise comes from the processing of the image, and it can be either separable like the additive noise pattern in image watermarking or nonseparable like the quantization noise in image coding. An application of the algorithm is demonstrated in the paper by using it to enhance image coders. Images decoded from the SNS incorporated coders have superior perceived quality than those without using SNS.

We have developed and demonstrated that spatial noise shaping (SNS) can enhance the performance of an image coder. SNS runs open-loop 2-D linear prediction (LP) in the frequency domain instead of in the time domain as compared to the standard LP used in speech and image coding. This predictive analysis/synthesis process over frequency possesses two important properties which result in a decoded image with superior quality compared to the one without using SNS. The first property is that since SNS can preserve the spatial structure of the image in the filter, those desirable details and sharp edges won't be damaged by the image coder and will be restored to the decoded image at the synthesis stage of SNS. The other unique property of SNS is that it effectively adapts the spatial structure of the quantization noise to that of the image roughness. Therefore, a more efficient use of masking effects can be exploited. In this paper we first explain how to use the noise shaping capability of SNS to hide the quantization noise in visually insensitive areas, such as textures and edges, but not in flat areas. Then, the techniques used to stabilize the 2-D LP filter are discussed. Finally, we show that SNS is applicable to either spectral domain coders or time domain coders, such as wavelets.

A spatial noise shaping (SNS) method based on human visual sensitivity is presented. It effectively shapes (or say hides) the noise of an image in areas which are not sensitive to human visual perception so that the resulting image is more pleasant to human eyes. The noise comes from the processing of the image, and it can be either separable like the additive noise pattern in image watermarking or non-separable like the quantization noise in image coding. An application of the algorithm is also demonstrated by using it to enhance image coders. Images decoded from the SNS incorporated coders possess a superior quality to those which do not use SNS.

We have studied and concluded that the time domain cross channel prediction is generally not applicable to perceptual audio coding. From the statistical analysis, correlation among certain channels seems to be enough to provide some coding gain and the prediction also successfully reduces the total energy of the signal to be coded. But this energy decrease and its resulting bit reduction mainly happens in the low frequency bands. In fact, the whitening effect of the prediction filter increases either the energy or the energy peaks in the high frequency region. As a result, more bits are required to code the high-frequency part of the signal and this increase outpaces the bit reduction realized in the low frequency region. Therefore, using cross channel linear prediction in the multichannel perceptual audio coding does not provide a net coding gain.

A perceptual audio coder typically consists of a filter-bank which breaks the signal into its frequency components. These components are then quantized using a perceptual masking model. Previous efforts have indicated that a high resolution filter-bank, e.g., the modified discrete cosine transform (MDCT) with 1024 subbands, is able to minimize the bit rate requirements for most of the music samples. The high resolution MDCT, however, is not suitable for the encoding of non-stationary segments of music. A long/short resolution or "window" switching scheme has been employed to overcome this problem but it has certain inherent disadvantages which become prominent at lower bit rates (<64 kbps for stereo). We propose a novel switched filter-bank scheme which switches between a MDCT and a wavelet filter-bank based on the signal characteristics. A tree structured wavelet filter-bank with properly designed filters offers natural advantages for the representation of non-stationary segments such as attacks. Furthermore, it allows for the optimum exploitation of perceptual irrelevancies.

In this paper we present a high quality audio coding system based on a novel nonuniform modulated filterbank coupled with time-varying cosine modulated filterbanks in a cascade architecture. The system makes use of psychoacoustic thresholds in a natural way to adapt the resolution of the filterbank to achieve high coding gain on a wide range of signal types. Results show that the system provides excellent quality at 64 kb/s and good quality at 48 kb/s for monophonic coding.

The notion of perceptual coding, which is based on the concept of distortion masking by the signal being compressed, is developed. Progress in this field as a result of advances in classical coding theory, modeling of human perception, and digital signal processing, is described. It is proposed that fundamental limits in the science can be expressed by the semiquantitative concepts of perceptual entropy and the perceptual distortion-rate function, and current compression technology is examined in that framework. Problems and future research directions are summarized.>

An audio filterbank coder using both monophonic and stereophonic redundancy and irrelevancy is described. This coder, with a transform filterbank, uses both signal-processing gains and perceptual gains to provide transparent or nearly transparent coding of CD and R-DAT audio signals at 192 kb/s/stereo pair, and high-quality coding at 128 kb/s/stereo pair. Methods of using the interchannel redundancies and psychoacoustic characteristics are described.>

The author reports on the extension of the entropy-coded perceptual transform coder (ECPTC) to stereo signals. The stereo ECPTC exploits both the redundancy in the stereo signal and the effects of acoustic mixing in the listening environment in order to encode the stereo signal at a bit rate much less than twice that for a monophonic signal. This coder uses the perceptual threshold developed by the author (IEEE J. Selected Areas in Commun. vol.6, no.2, p.314-23, Feb. 1988) along with a noiseless compression scheme to provide a bit-rate reduction of approximately one third compared to that for two independent monophonic channels. Typical rates available from the coder are 64 kb/s for a 6.6-kHz bandwidth, 14-kHz sampling rate transparent monophonic channel, 128 kb/s for a 15-kHz bandwidth, 32-kHz sampling rate stereo channel with FM or better quality, and 192 kb/s for a 20-kHz bandwidth, 44.1-kHz sampling rate stereo channel with perceived quality equal to that of the original signal. The 64-kb/s, 128-kb/s, and 192-kb/s rates correspond to ISDN (integrated services digital network) 1B, 2B, and 3B rates respectively, suggesting applications within the ISDN framework.>

The authors present a 16-band subband coder arranged as four equal-width subbands in each dimension, It uses an empirically derived perceptual masking model, to set noise-level targets not only for each subband but also for each pixel in a given subband. The noise-level target is used to set the quantization levels in a DPCM (differential pulse code modulation) quantizer. The output from the DPCM quantizer is then encoded, using an entropy-based coding scheme, in either 1*1, 1*2, or 2*2 pixel blocks. The type of encoding depends on the statistics in each 4*4 subblock of a particular subband. One set of codebooks, consisting of less than 100000 entries, is used for all images, and the codebook subset used for any given image is dependent on the distribution of the quantizer outputs for that image. A block elimination algorithm takes advantage of the peaky spatial energy distribution of subbands to avoid using bits for quiescent parts of a given subband. Using this system, high-quality output is obtainable at bit rates from 0.1 to 0.9 bits/pixel, and nearly transparent quality requires 0.3 to 1.5 bits/pixel.>

The perceptual entropy of each short-term section of the audio stimuli is estimated as the number of bits required to encode the short-term spectrum of the signal to the resolution measured by this process provide an entropy estimate, for transparent coding, of 1.4 (mean) or 2.1 (peak) bits/sample for telephone speech (200-3200-Hz bandwidth sampled at 8 kHz). The entropy measures for audio signals of other bandwidths and sampling rates is also reported.>

A 4-b/sample transform coder is designed using a psychoacoustically derived noise-making threshold that is based on the short-term spectrum of the signal. The coder has been tested in a formal subjective test involving a wide selection of monophonic audio inputs. The signals used in the test were of 15-kHz bandwidth, sampled at 32 kHz. The bit rate of the resulting coder was 128 kb/s. The subjective test shows that the coded signal could not be distinguished from the original at that bit rate. Subsequent informal work suggests that a bit rate of 96 kb/s may maintain transparency for the set of inputs used in the test.>

The Analog Voice Privacy System is based on individual sample permutation of the output samples of a sub-band coder analysis filterbank. The system has a large number of digital keys, giving it the strength of a digital encryption system, but also retains the good quality characteristics of analog scramblers. It has been implemented in a real-time hardware prototype designed for evaluation in the field. The units work with any modular telephone and standard 120 volts AC electricity. The device contains two circuitry boards, one for analog and one for digital processing which contain four digital signal processors. There are 125! possible permutation keys. These prototypes were designed to be tested in real telephone environments. To date, the device has been successfully tested over long distance telephone connections, several different analog and digital PBXs and telephone switches, and a channel simulator. The quality of the decrypted speech is considered very natural, and in particular, speaker recognition is retained. This is a significant advantage over digital vocoders. This paper describes the underlying principles of the algorithm, the details of its implementation and laboratory test results.

Development of Circuit Switched Digital Capability (CSDC) in the United States and the general evolution toward an integrated digital services network worldwide has created interest in the development of 56 or 64 kbps digital coders for audio signals with 6 or 7 kHz bandwidth. To date there have been a number of coders proposed but no formal testing of them. Furthermore, a method for testing them has not been proposed. In this paper we discuss the problem of testing such coders and describe a series of both objective and subjective tests which can be performed using a real-time coding system. We discuss the results of these tests when performed on a selection of coders which we have implemented.

A digital speech coder has been designed for real-time operation for a data rate of 9.6 kb/s. The design is based on a combination of two speech compression techniques: Time-Domain Harmonic Scaling (TDHS) and Sub-Band Coding (SBC). It is a highly modularized hardware implementation using five Bell Laboratories Digital Signal Processor (DSP) integrated circuits as the key processing elements. Three DSPs are used in the encoder for pitch detection, TDHS compression and sub-band encoding. Another two DSPs are used in the receiver for sub-band decoding and TDHS expansion.

This paper reviews our recent efforts in the design and implementation of real-time speech coders. We discuss our approach and methodology for real-time hardware for coder techniques ranging from low to high complexity. Examples of realizations are given for each approach. They include adaptive differential PCM coding, subband coding, harmonic scaling with subband coding, and adaptive transform coding. Low to medium complexity techniques are based on the use of the Bell Laboratories digital signal processing (DSP) integrated circuit. High complexity block processing techniques are based on the use of an array processing computer. We conclude with an assessment of the performance versus complexity tradeoffs involved in these coding methods.

We describe digital encoding and automatic silenceediting techniques for PCM and ADPCM voice storage in the MC68000 microprocessor. We give program details for encoding, silence elimination, storage of active speech bursts and reconstitution of the signal upon read-back. Coding rates in the range 64 down to 16 kbits/s are implemented. We find that silent intervals corresponding, typically, to 20-40 percent of the duration of continuous input speech can be usefully eliminated. Available storage is thereby used more efficiently. The eliminated intervals are automatically reinserted upon read-back. We further demonstrate automatic scaling of the silent intervals in the reconstituted output. We confirm that a variation of ± 50 percent in the reinserted silent intervals is typically acceptable. This latitude can significantly alleviate buffer requirements in a packet transmission system for voice.

In this paper we discuss the implementation of sub-band coding (SBC) on a programmable digital signal processing (DSP) integrated circuit developed at Bell Laboratories. A two-band SBC has been developed for use in 7 kHz bandwidth transmission (AM broadcast quality) at a bit-rate of 56 kb/s. A four-band SBC has been developed for use in speech coding (3.2 kHz bandwidth) at bit rates ranging from 16 kb/s to 24 kb/s. We first review the capabilities of the DSP integrated circuit. We then discuss the hardware and software development environment used for programming real-time SBC on the DSP. Next we discuss block diagrams of the SBC algorithms that were implemented. Finally, we discuss aspects of how these algorithms were programmed in real-time on the DSP and features of the DSP that are utilized.

This paper discusses a family of filters that have been designed for Quadrature Mirror Filter (QMF) Banks. These filters provide a significant improvement over conventional optimal equiripple and window designs when used in QMF banks. The performance criterion for these filters differ from those usually used for filter design in a way which makes the usual filter design techniques difficult to apply. Two filters are actually designed simultaneously, with constraints on the stop band rejection, transition band width, and pass and transition band performance of the QMF filter structure made from those filters. Unlike most filter design problems, the behavior of the transition band is constrained, which places unusual requirements on the design algorithm. The requirement that the overall passband behavior of the QMF bank be constrained (which is a function of the passband and stop band behavior of the filter) also places very unusual requirements on the filter design. The filters were designed using a Hooke and Jeaves optimization routine with a Hanning window prototype. Theoretical results suggest that exactly flat frequency designs cannot be created for filter lengths greater than 2, however, using the discussed procedure, one can obtain QMF banks with as little as ±.0015dB ripple in their frequency response. Due to the nature of QMF filter applications, a small set of filters can be derived which will fit most applications.

This paper explores the use of slope limiting filters to mitigate the errors induced in adaptive and non-adaptive PCM and DPCM speech. The two correctors discussed use slope limiting to remove the spikes associated with bit errors. They are treated as add-on systems with no knowledge of the input signal other than that it is a speech waveform, and use very simple replacement algorithms when they detect an anomalous condition. The methods do not make an "optimum" replacement of anomalous conditions; however, simple replacement algorithms result in significant improvement of corrupted speech, especially where listening fatigue is a factor. They use inexpensive (analog and delta modulation) circuits which are easily constructed and installed.

A two-band coding system has been constructed for the purpose of providing commentary grade (7 kHz bandwidth) speech or music transmission at 56 or 64 kbits/s. The lower band, 0-to-3650 Hz, is coded with 4 bit ADPCM and the upper band, 3600-to-6800 Hz, is coded with 3 bit or 4 bit/sample APCM. The quality of the coded signal makes the method useful for news and sports broadcasts, and possibly for AM remote music broadcasting. The audio sounds better than that produced by two conventional alternatives: 3200 Hz bandwidth with 8-bit/sample coding and 7000 Hz bandwidth with a single 4-bit/sample coder. The sample may be used in any place with access to a 56 kbit/s Dataphone Digital Service port or to other 56 or 64 kbit/s lines. The power consumption is approximately 12 W in the present form; it could be reduced by a factor of at least two by hardware optimization.

A two-band coding system has been constructed for the purpose of providing commentary grade (7 kHz bandwidth) speech or music transmission at 56 or 64 kb/s. The lower band, 0 to 3650 Hz, is coded with 4 bit adpcm and the upper band, 3600 - 6800 Hz, is coded with 3 bit or 4 bit apcm. The quality of the coded signal makes the method useful for news and sports broadcasts, and possibly for AM remote music broadcasting. The audio sounds better than that produced by two conventional alternatives: 3200 Hz bandwidth with 8-bit coding and 7000 Hz bandwidth with a single 4-bit coder. The system may be used in any place with access to a 56 kb/s Dataphone Digital Service port or to other 56 or 64 kb/s lines. The power consumption is approximately 12 watts in the present form; it could be reduced by a factor of at least 2 by hardware optimization.

We describe an adaptive coding system capable of representing audio frequency signals in a wide variety of digital code formats. To provide adaptive quantization, the step size is adjusted by voltage controlled amplifiers operating with analog control signals. Sine wave signal-to-noise ratio measurements demonstrate important relationships among the main code classifications: pcm, apcm, dpcm, adpcm. The multipurpose coder has proved a useful laboratory aid in the design of special purpose coders. After adjustment and evaluation of parameters using the multipurpose coder one proceeds to an economical, specialized design.