With increasing computation power, network bandwidth, and improvements in display and capture technologies, fully immersive conferencing and tele-immersion is becoming ever closer to reality. Outside of video, one of the key components needed is high quality spatialized audio. This paper presents an implementation of a relatively low complexity, simple solution which allows realistic audio spatialization of arbitrary positions in a 3D video conference. When combined with pose tracking, it also allows the audio to change relative to which position on the screen the viewer is looking at.

Audio spatialization is becoming an important part of creating realistic experiences needed for immersive video conferencing and gaming. Using a combined head and room impulse response (CHRIR) has been recently proposed as an alternative to using separate head related transfer functions (HRTF) and room impulse responses (RIR). Accurate measurements of the CHRIR at various source and listener locations and orientations are needed to perform good quality audio spatialization. However, it is infeasible to accurately measure or model the CHRIR for all possible locations and orientations. Therefore, low-complexity and accurate interpolation techniques are needed to perform audio spatialization in real-time. In this paper, we present a frequency domain interpolation technique which naturally interpolates the interaural level difference (ILD) and interaural time difference (ITD) for each frequency component in the spectrum. The proposed technique allows for an accurate and low-complexity interpolation of the CHRIR as well as allowing for a low-complexity audio spatialization technique which can be used for both headphones as well as loudspeakers.

It has been proven that spatial audio enhances the realism of sound for teleconferencing. Previously, solutions have been proposed for multiparty conferencing where each remote participant is assumed to have his/her own microphone, and for conferencing between two rooms where the microphones in one room are connected to the equal number of loudspeakers in the other room. Either approach has its limitations. Hence, we propose a new scheme to improve stereophonic conferencing experience through an innovative use of microphone arrays. Instead of operating in the default mode where a single channel is produced using spatial filtering, we propose to transmit all channels forming a collection of spatial samples of the sound field. Those samples are warped appropriately at the remote site, and are spatialized together with audio streams from other remote sites if any, to produce the perception of a virtual sound field. Real-world audio samples are provided to showcase the proposed technique. The informal listening test shows that majority of the users prefer the new experience.

This paper describes a method for designing oversampled DFT filter banks (FB) optimized for subband acoustic echo cancellation (AEC). For this application, the design requirements typically are good echo cancellation quality, low delay, small reconstruction error, and low computation complexity. Our method explicitly includes a model for echo return loss enhancement (ERLE) as part of the optimization criteria. Convergence of the high-dimensional, nonlinear optimization problem is facilitated by decorrelating the prototype filter impulse response via a discreet cosine transform (DCT), discarding many insignificant coefficients and thus reducing the dimensionality of the search. The experimental results demonstrate the effectiveness of our design and the effects on the performance of AEC, with ELRE improvements on the order of 3 dB or better. The method is flexible and could also be extended to other application domains.

Audio coding at low bitrates suffers from artifacts due to spectrum truncation. Typical audio codecs code multi-channel sources using transforms across the channels to remove redundancy such as middle (mid) - side (M/S) coding. At low bitrates, the spectrum of the coded channels is truncated and the spectrum of the channels with lower energy, such as the side channel, is truncated severely, sometimes entirely. This results in a muffled sound due to truncation of all coded channels beyond a certain frequency. It also results in a loss of spatial image even at low frequencies due to severe truncation of the side channel. Previously we have developed a low bitrate coding method to combat the loss of higher frequencies caused by spectrum truncation. In this paper, we present a novel low bitrate audio coding scheme to mitigate the loss of spatial image. Listening tests show that the combination of the two low bitrate coding methods results in a audio codec that can get good quality even at bitrates as low as 32kbps for stereo content with low decoder complexity.

It is known that during multi-party conferencing spatialized audio which maps remote participants' voices to distinct virtual locations improves the listening experience. In this paper, we consider the case when the audio is rendered through headphones due to e.g. privacy reasons. Although existing headphone spatial audio techniques abound, most lack the desired realism dictated by listeners' expectation of naturalness in audio conferencing. In light of the situation, we propose a novel approach of spatial audio processing for headphones, where we measure the combined head and room impulse responses (CHRIRs) in an actual physical setting which are then directly used to spatialize remote participants' voices. Through proper processing of the CHRIRs, our solution is able to offer a higher degree of realism, closely approximating binaural recordings. We note that the success, however, comes with certain limitations and presented solutions to mitigate the shortcomings of the proposed technique. As a result, a user can adapt the source localization and the degree of reverberation to her own subjective preferences. We also show that the computation load can be effectively reduced and becomes very reasonable for modern computer hardware.

The use of multiple entropy models for Huffman or arithmetic coding is widely used to improve the compression efficiency of many algorithms when the source probability distribution varies. However, the use of multiple entropy models increases the memory requirements of both the encoder and decoder significantly. In this paper, we present an algorithm which maintains almost all of the compression gains of multiple entropy models for only a very small increase in memory over one which uses a single entropy model. This can be used for any entropy coding scheme such as Huffman or arithmetic coding. This is accomplished by employing multiple entropy models only for the most probable symbols and using fewer entropy models for the less probable symbols. We show that this algorithm reduces the audio coding bitrate by 5%–8% over an existing algorithm which uses the same amount of table memory by allowing effective switching of the entropy model being used as source statistics change over an audio transform block.

Audio coding at low bitrates typically suffers from artifacts caused by bandwidth truncation. In this paper we present a novel scheme to code audio signals at low bitrates which uses a traditional scalar quantization followed by entropy coding to code some portions of the spectrum (typically the lower portion). The other portions (typically the higher portions) of the spectrum are coded at a low bitrate using an adaptive gain shape vector quantizer where the codebook for vector quantization is formed by unmodified or modified versions of the portions of the spectrum which have already been coded. Fixed pre-trained codebooks are also available for use in certain cases. The use of such a scheme results in an audio codec which has been shown to be among the best audio codecs available at low bitrates. In addition, the decoder complexity of this audio codec is significantly lower than any other codec of equal quality at low bitrates.

Motion discontinuities arise when there are occlusions or multiple moving objects in the scene that is imaged. Conventional regularization techniques use smoothness constraints but are not applicable to motion discontinuities. In this paper, we show that discontinuous (or multiple) motion estimation can be viewed as a multicomponent harmonic retrieval problem. From this viewpoint, a number of established techniques for harmonic retrieval ran be applied to solve the challenging problem of discontinuous (or multiple) motion. Compared with existing techniques, the resulting algorithm is not iterative, which not only implies computational efficiency but also obviates concerns regarding convergence or local minima. It also adds flexibility to spatio-temporal techniques which have suffered from lack of explicit modeling of discontinuous motion. Experimental verification of our framework on both synthetic data as well as real image data is provided.

This paper introduces a new technique in block-based motion estimation and motion compensation for object-based video coding. Out of many approaches for object-based coding, hybrid algorithms that take advantages of the traditional motion compensated block-based DCT approach are noted for their maturity and simplicity. Indeed, it has become the main feature of the upcoming international video standard MPEG-4. In this type of hybrid algorithm, the boundary of the objects calls for special attention. The discontinuity at object boundary could be very costly if not handled properly. We discovered that efficient motion estimation and compensation for the object boundary is most critical to the performance of object-based video coding. Our studies show that padding, a process of boundary value extension, is essential in motion estimation/compensation of arbitrarily shaped objects. We have designed novel padding algorithms and report our preliminary results in this paper.

A layered video object coding system is presented in this paper. The goal is to improve video coding efficiency by exploiting the layering of video and to support content-based functionality. These two objectives are accomplished using a sprite technique and an affine motion model on a per-object basis. Several novel algorithms have been developed for mask processing and coding, trajectory coding, sprite accretion and coding, locally affine motion compensation, error signal suppression, and image padding. Compared with conventional frame-based coding methods, better experimental results on both hybrid and natural scenes have been obtained using our coding scheme. We also demonstrate content-based functionality which can be easily achieved in our system.

Motion smear is an important visual cue for motion perception by the human vision system (HVS). However, in image analysis research, exploiting motion smear has been largely ignored. Rather, motion smear is usually considered as a degradation of images that needs to be removed. In this paper, the authors establish a computational model that estimates image motion from motion smear information-"motion from smear". In many real situations, the shutter of the sensing camera must be kept open long enough to produce images of adequate signal-to-noise ratio (SNR), resulting in significant motion smear in images. The authors present a new motion blur model and an algorithm that enables unique estimation of image motion. A prototype sensor system that exploits the new motion blur model has been built to acquire data for "motion-from-smear". Experimental results on images with both simulated smear and real smear, using the authors' "motion-from-smear" algorithm as well as a conventional motion estimation technique, are provided. The authors also show that temporal aliasing does not affect "motion-from-smear" to the same degree as it does algorithms that use displacement as a cue. "Motion-from-smear" provides an additional tool for motion estimation and effectively complements the existing techniques when apparent motion smear is present.

Image motion estimation using the spatiotemporal approach has largely relied on the constant velocity assumption, and thus becomes inappropriate when the velocity of the imaged scene or the camera changes during the data acquisition time. Using a polynomial or a trigonometric polynomial model for the time variation of the image motion, spatiotemporal algorithms are developed in this paper to handle time-varying (but space-invariant) motion. Under these models, it is shown that time-varying image motion estimation is equivalent to parameter estimation of one-dimensional (1-D) polynomial phase or phase-modulated signals, which allows one to exploit well-established results in radar signal processing. When compared with alternative approaches, the resulting motion estimation algorithms produce more accurate estimates. Simulation results are provided to demonstrate the proposed schemes.

The authors are interested in estimating the Doppler shift occurred in weather radar returns, which yields precipitation velocity information. Conventional techniques including the pulse pair processor rely heavily on the assumption that the additive noise is white and hence their performance degrades when the noise color is unknown. Because the data length for a given range gate is usually small, the authors employ the high resolution MUSIC algorithm to estimate the Doppler shift. The challenge lies not only in proving that MUSIC is applicable to weather radar signals which are affected by multiplicative noise, but also in showing that MUSIC is robust when the additive noise is colored. The resulting algorithm can also be used to infer wind speed from a small number of lidar observations where the velocity is approximately constant. Assuming linear shear over a longer range, they employ the ambiguity function to estimate the acceleration and instantaneous wind velocity. Real weather radar and lidar data as well as simulated examples are provided to illustrate the performance of the algorithms.

Motion discontinuities arise when there are occlusions or multiple moving objects in the scene that is imaged. We show that discontinuous (or multiple) motion estimation can be viewed as a multicomponent harmonic retrieval problem. Compared with existing techniques, the resulting algorithm is not iterative, which not only implies computational efficiency but also obviates concerns regarding convergence or local minima. It also adds flexibility to spatio-temporal techniques which have suffered from lack of explicit modeling of discontinuous motion.

Image motion estimation usually relies on using interframe displacement as the information cue for motion. Another cue has been proposed, namely motion smear. An algorithm used to compute the image motion from the smear information ("motion-from-smear") was proposed by Chen, Nandhakumar and Martin (see Proc. IEEE Intern. Conf. Image Proc., vol.III, p.761-765, Austin, Texas, November 1994) where the algorithm was verified with simulated motion smeared images. This paper extends the work of Chen et al. by presenting (1) experimental results on real data, (2) a feasible sensor design and (3) a generalization to space-varying motion.

Image motion estimation using the spatio-temporal approach has largely relied on the constant velocity assumption. Spatio-temporal algorithms are developed in this paper to handle time-varying motion. Time-variation is modeled either as a polynomial or as a periodic function of time. Under these models, it is shown that time-varying image motion estimation is equivalent to parameter estimation of one-dimensional polynomial phase or phase modulated signals. Simulation results are provided to illustrate the proposed algorithm.>

Motion smear is an important visual cue for the human vision system (HVS) to perceive motion. However, in image analysis research, using motion smear as a valuable visual cue for motion estimation has remained largely ignored. Rather, motion smear is usually considered as a degradation on images that need to be removed. A computational model has been established to estimate image motion from motion smear information. Both theoretical derivation and experimental results are provided.>