In the history of audio and acoustic signal processing, perceptual audio coding has certainly excelled as a bright success story by its ubiquitous deployment in virtually all digital media devices, such as computers, tablets, mobile phones, set-top-boxes, and digital radios. From a technology perspective, perceptual audio coding has undergone tremendous development from the first very basic perceptually driven coders (including the popular mp3 format) to today’s full-blown integrated coding/rendering systems. This paper provides a historical overview of this research journey by pinpointing the pivotal development steps in the evolution of perceptual audio coding. Finally, it provides thoughts about future directions in this area.

Efficient audioquality assessment is vital for streamlining audio codec development. Objective assessment tools have been developed over time to algorithmically predict quality ratings from subjective assessments, the gold standard for quality judgment. Many of these tools use perceptual auditory models to extract audio features that are mapped to a basic audio quality score prediction using machine learning algorithms and subjective scores as training data. However, existing tools struggle with generalization in quality prediction, especially when faced with unknown signal and distortion types. This is particularly evident in the presence of signals coded using non-waveform-preserving parametric techniques. Addressing these challenges, this two-part work proposes extensions to the Perceptual Evaluation of Audio Quality (PEAQ - ITU-R BS.1387-1) recommendation. Part 1 focuses on increasing generalization, while Part 2 targets accurate spatial audio quality measurement in audio coding. To enhance prediction generalization, this paper (Part 1) introduces a novel machine learning approach that uses subjective data to model cognitive aspects of audio quality perception. The proposed method models the perceived severity of audible distortions by adaptively weighting different distortion metrics. The weights are determined using an interaction cost function that captures relationships between distortion salience and cognitive effects. Compared to other machine learning methods and established tools, the proposed architecture achieves higher prediction accuracy on large databases of previously unseen subjective quality scores. The perceptually-motivated model offers a more manageable alternative to general-purpose machine learning algorithms, allowing potential extensions and improvements to multi-dimensional quality measurement without complete retraining.

Objective audio quality measurement systems often use perceptual models to predict the subjective quality scores of processed signals, as reported in listening tests. Most systems map different metrics of perceived degradation into a single quality score predicting subjective quality. This requires a quality mapping stage that is informed by real listening test data using statistical learning (i.e., a data-driven approach) with distortion metrics as input features. However, the amount of reliable training data is limited in practice, and usually not sufficient for a comprehensive training of large learning models. Models of cognitive effects in objective systems can, however, improve the learning model. Specifically, considering the salience of certain distortion types, they provide additional features to the mapping stage that improve the learning process, especially for limited amounts of training data. We propose a novel data-driven salience model that informs the quality mapping stage by explicitly estimating the cognitive/degradation metric interactions using a salience measure. Systems incorporating the novel salience model are shown to outperform equivalent systems that only use statistical learning to combine cognitive and degradation metrics, as well as other well-known measurement systems, for a representative validation dataset.

The term “immersive audio” is frequently used to describe an audio experience that provides the listener the sensation of being fully immersed or “present” in a sound scene. This can be achieved via different presentation modes, such as surround sound (several loudspeakers horizontally arranged around the listener), 3D audio (with loudspeakers at, above, and below listener ear level), and binaural audio to headphones. This article provides an overview of two recent standards that support the bitrate-efficient carriage of high-quality immersive sound. The first is MPEG-H 3D audio, which is a versatile standard that supports multiple immersive sound signal formats (channels, objects, and higher order ambisonics) and is now being adopted in broadcast and streaming applications. The second is MPEG-I immersive audio, an extension of 3D audio, currently under development, which is targeted for virtual and augmented reality applications. This will support rendering of fully user-interactive immersive sound for three degrees of user movement [three degrees of freedom (3DoF)], i.e., yaw, pitch, and roll head movement, and for six degrees of user movement [six degrees of freedom (6DoF)], i.e., 3DoF plus translational x, y, and z user position movements.

Over the past few decades, computational methods have been developed to estimate perceptual audio quality. These methods, also referred to as objective quality measures, are usually developed and intended for a specific application domain. Because of their convenience, they are often used outside their original intended domain, even if it is unclear whether they provide reliable quality estimates in this case. This work studies the correlation of well-known state-of-the-art objective measures with human perceptual scores in two different domains: audio coding and source separation. The following objective measures are considered: fwSNRseg, dLLR, PESQ, PEAQ, POLQA, PEMO-Q, ViSQOLAudio, (SI-)BSSEval, PEASS, LKR-PI, 2f-model, and HAAQI. Additionally, a novel measure (SI-SA2f) is presented, based on the 2f-model and a BSSEval-based signal decomposition. We use perceptual scores from 7 listening tests about audio coding and 7 listening tests about source separation as ground-truth data for the correlation analysis. The results show that one method (2f-model) performs significantly better than the others on both domains and indicate that the dataset for training the method and a robust underlying auditory model are crucial factors towards a universal, domain-independent objective measure.

The Perceptual Evaluation of Audio Quality (PEAQ) method as described in the International Telecommunication Union (ITU) recommendation ITU-R BS.1387 has been widely used for computationally estimating the quality of perceptually coded audio signals without the need for extensive subjective listening tests. However, many reports have highlighted clear limitations of the scheme after the end of its standardization, particularly involving signals coded with newer technologies such as bandwidth extension or parametric multi-channel coding. Until now, no other method for measuring the quality of both speech and audio signals has been standardized by the ITU. Therefore, a further investigation of the causes for these limitations would be beneficial to a possible update of said scheme. Our experimental results indicate that the performance of PEAQ's model of disturbance loudness is still as good as (and sometimes superior to) other state-of-the-art objective measures, albeit with varying performance depending on the type of degraded signal content (i.e. speech or music). This finding evidences the need for an improved cognitive model. In addition, results indicate that an updated mapping of Model Output Values (MOVs) to PEAQ's Distortion Index (DI) based on newer training data can greatly improve performance. Finally, some suggestions for the improvement of PEAQ are provided based on the reported results and comparison to other systems.

This work introduces a feature extracted from stereophonic/binaural audio signals aiming to represent a measure of perceived quality degradation in processed spatial auditory scenes. The feature extraction technique is based on a simplified stereo signal model considering auditory events positioned towards a given direction in the stereo field using amplitude panning (AP) techniques. We decompose the stereo signal into a set of directional signals for given AP values in the Short-Time Fourier Transform domain and calculate their overall loudness to form a directional loudness representation or maps. Then, we compare directional loudness maps of a reference signal and a deteriorated version to derive a distortion measure aiming to describe the associated perceived degradation scores reported in listening tests.The measure is then tested on an extensive listening test database with stereo signals processed by state-of-the-art perceptual audio codecs using non waveform-preserving techniques such as bandwidth extension and joint stereo coding, known for presenting a challenge to existing quality predictors. Results suggest that the derived distortion measure can be incorporated as an extension to existing automated perceptual quality assessment algorithms for improving prediction on spatially coded audio signals.

While the human hearing capability for horizontally localized sound sources is mostly based on binaural cues, the localization of elevation along the "cones of confusion" can only rely on spectral cues provided by the directionality of head related transfer functions (HRTF). This paper explores how the magnitude of spectral differences between HRTF affects the availability of spectral cues to perceive and detect different elevations and hence limits localization accuracy. A method to predict localization accuracy is presented, based on analysis of the gradient of HRTF log-spectral differences for elevation positional changes. To this end, the localization accuracy is estimated from an HRTF database with 45 subjects. Validation of the results shows good agreement with results from subjective localization experiments reported in literature.

High Resolution Envelope Processing (HREP) is a new tool for improved perceptual coding of audio signals that predominantly consist of many dense transient events, such as applause, rain drop sounds, etc. These signals have traditionally been very difficult to code for perceptual audio codecs, particularly at low bit rates. Based on the gain control principle, HREP acts as a pre-/post-processor pair to perceptual audio codecs and preserves the temporal fine structure and subjective quality of applause-like signals. Subjective tests have shown a significant improvement in audio quality of around 12 MUSHRA points by HREP processing at 48 kbps stereo when used together with an MPEG-H 3D Audio codec. The new coding tool has been adopted as part of MPEG-H 3D Audio Second Edition.

A common problem in audio signal processing is to mix two or more signals into one sum signal. The mixing procedure, known as down-mixing, usually introduces some signal impairments, especially if two signals contain similar but phase shifted signal components. Summing up such signals results in severe comb-filter artifacts. In this paper, we propose a novel down-mix method which prevents comb-filter effects. This is achieved by suppressing the coherent signal parts of one input signal prior to mixing. During the down-mixing, a scaling gain is applied ensuring the preservation of overall signal energy. Furthermore, a phase-align extension to the down-mixer is introduced. The proposed down-mixer is evaluated qualitatively in comparison with two existing down-mix approaches as well as quantitatively by means of a distortion analysis.

This paper introduces high-quality audio coding using psychoacoustic models. This technology is now abundant, with gadgets named after a standard (mp3 players) and the ability to play high-quality audio from literally billions of devices. The usual paradigm for these systems is based on filterbanks, followed by quantization and coding, controlled by a model of human hearing. The paper describes the basic technology, theoretical framework to apply to check for optimality, and the most prominent standards built on the basic ideas and newer work.

Subband modulation spectra (SB-MS) have proven to be useful for many applications in audio signal analysis and audio signal processing. This paper presents a novel blind watermarking scheme for embedding and detecting a watermark in the subband modulation spectral domain representation of an audio signal. Some results of audio quality, data rate and robustness are presented. The scheme is shown to be robust to a number of attacks while providing excellent subjective audio quality.

Acoustic echo cancellation (AEC) has become an essential and well-known enabling technology for hands-free communication and human-machine interfaces. AEC for two or more reproduction channels aims at identifying the echo paths between the microphone and each audio reproduction source in order to cancel the associated echo contribution. A number of preprocessing methods have been proposed to decorrelate stereo audio signals in order to enable an unambiguous identification of each echo path and to thus ensure robustness to changing sound source locations. While several of these methods provide enough decorrelation to achieve proper AEC convergence in the stereo case, considerations of subjective sound quality have frequently not been addressed adequately. This paper compares the performance of several methods in terms of both convergence speed and aspects of sound perception, and proposes a novel signal decorrelation approach with attractive properties. The superior performance of the proposed method is demonstrated for 5.1 surround sound reproduction.

In the area of low bitrate audio coding, the emergence of "spatial audio coding" (SAC) technology is one of the most remarkable innovations during the recent years. By exploiting the human perception of spatial sound, these coding schemes are capable of transmitting high quality surround sound using bitrates that have been used so far for carrying traditional two-channel stereo audio. After the recent finalization of the MPEG surround (MPS) specification, a next technology generation is envisaged for standardization within ISO/MPEG allowing bitrate-efficient and backward compatible coding of several sound objects. On the receiving side, such a "spatial audio object coding" (SAOC) system renders the objects interactively into a sound scene on a reproduction setup of choice. The paper reviews the principles and current status of SAC schemes and discusses their evolution towards SAOC.

Recently lifting-based integer transforms have received much attention, especially in the area of lossless audio and image coding. The usual approach is to apply the lifting scheme to each Givens rotation. Especially in the case of long transform sizes in audio coding applications, this leads to a considerable approximation error in the frequency domain. This paper presents a multidimensional lifting approach for reducing this approximation error. In this approach, large parts of the transform are calculated without rounding operations, only the output is rounded and added. The new approach is applied and evaluated for both the integer modified discrete cosine transform (IntMDCT) and the integer fast Fourier transform (IntFFT).

This papers presents an embedded fine grain scalable perceptual and lossless audio coding scheme. The enabling technology for this combined perceptual and lossless audio coding approach is the integer modified discrete cosine transform (IntMDCT), which is an integer approximation of the MDCT based on the lifting scheme. It maintains the perfect reconstruction property and therefore enables efficient lossless coding in the frequency domain. The close approximation of the MDCT also allows us to build a perceptual coding scheme based on the IntMDCT. In this paper a bitsliced arithmetic coding technique is applied to the IntMDCT values. Together with the encoded shape of the masking threshold a perceptually hierarchical bitstream is obtained, containing several stages of perceptual quality and extending to lossless operation when transmitted completely. A concept of encoding subslices is presented in order to obtain a fine adaptation to the masking threshold especially in the range of perceptually transparent quality.

The Modified Discrete Cosine Transform (MDCT) is widely used in modem perceptual audio coding schemes. In this paper we present an integer approximation of this lapped transform, called IntMDCT, which is derived from the MDCT using the lifting scheme. This reversible integer transform inherits most of the attractive properties of the MDCT, exhibiting a good spectral representation of the audio signal, critical sampling and overlapping of blocks. This makes the IntMDCT well suited for both lossless audio coding as well as for combined perceptual and lossless audio coding. A scalable system is presented providing a lossless enhancement of perceptual audio coding schemes, such as MPEG-2 AAC.