Binaural sound capture and reproduction can produce sound of remarkable quality and realism over headphones. It also suffers from shortcomings that have hindered its widespread usage. Most notably, it does not eliminate the in-head localization that is also experienced with stereo recordings. Motion-tracking that restores the dynamic cues caused by normal movements of the head is necessary to externalize binaural sounds, but its computational requirements may be heavy. We have developed practical methods that realize motion-tracked binaural sound (MTB) both in capture and reproduction. In this paper, we review briefly the research on the effect of head motion on the perception of spatial sound, describe MTB and its variants, compare it to HTRF-based rendering, and discuss our experience with binaural motion tracked sound.

While mobile technology precludes large electronic displays for visual immersion, sound heard over headphones - a widely accepted technology - is ideally suited for mobile applications. In this paper, we report on a newly developed immersive headphone-based approach that opens new opportunities for mobile multimedia. This new motion-tracked binaural sound technology (abbreviated as MTB) provides and exploits the strong perceptual cues that are created by the voluntary motions of the listener's head. A head-tracker is used to modify dynamically the sound heard by the listener, recreating the natural experience. After presenting an overview of the key concepts and of the challenges of the implementation of MTB, we describe examples of MTB spatial sound applications. Finally, we outline new mobile multimedia applications that would combine immersive spatial sound, head-tracking, and small visual displays.

[Summary form only given]. A wavelet coding method for arbitrarily shaped image data, applicable to object-oriented coding of moving pictures, and to the efficient representation of texture data in computer graphics is proposed. The wavelet transform of an arbitrarily shaped image is obtained by applying the symmetrical extension technique at region boundaries and keeping the location of the wavelet coefficient. For entropy coding of the wavelet coefficients, the zerotree coding technique is modified to work with arbitrarily shaped regions by treating missing (outside of the decomposed support) coefficients as insignificant and transmitting only those zerotree symbols which are in the decomposed support. The coding performance of the proposed method for several test images that include a person, a teapot and a necklace is compared to a shape-adaptive DCT and an ordinary DCT method applying low pass extrapolation to the DCT block containing the region boundaries. Experiments show that the proposed method has a better coding efficiency compared to SA-DCT and the ordinary DCT method.

Experimentally measured head-related transfer functions reveal that the interaural time delay varies from person to person. Furthermore, it is not constant around a cone of confusion, but can vary by as much as 18% of the maximum interaural delay. The major sources for this variation are shown to be the shape of the head and the displacement of the ears from the center of the head. A simple ellipsoidal head model is presented that can accurately account for this ITD variation and can be adapted to individual listeners.

This paper shows a picture quality evaluation model for JPEG coded color images. This methodology is obtained by a multi-variable analysis of picture distortion and it provides a numerical value of picture quality. The accuracy of the picture quality evaluation is dependent upon characteristics of the picture quality evaluation models. The error components e(x,y) are computed by subtracting color components (RGB, CIE L*a*b*, L*u*v*, etc.) of distorted compressed images from the original ones. The evaluation model is obtained by multi-variable analysis of distortion models. We then use regression methods to combine these picture quality evaluation models into a single number representative of the quality of a given image. We also evaluate the characteristics of picture quality evaluation models, because good picture quality evaluation models are indispensable to estimate the subjective mean opinion score (MOS) with a high accuracy. This model evaluates picture quality with high accuracy and estimates MOS well.

We present an algorithm to greatly reduce the directionality bias inherent in existing finite difference implementations of the non-linear anisotropic diffusion equation when applied to images. We demonstrate the merits of the algorithm by examining its scale space and noise removal properties.

In this paper, we propose an approach to improve the noise removal/structure preservation properties of non-linear, inhomogenous diffusion filtering algorithms. These algorithms are locally driven by the gradient vector. We introduce an algorithm to obtain an estimate of the gradient vector field using local and global information contained in the original noisy gradient field. The estimated gradient field is then applied to a linear inhomogeneous diffusion process. Good structure preservation/image denoising properties are obtained using this new approach.

A new methodology for the determination of an objective metric for still image coding is reported. This methodology is applied to obtain a picture quality scale (PQS) for the coding of achromatic images over the full range of image quality defined by the subjective mean opinion score (MOS). This PQS takes into account the properties of visual perception for both global features and localized disturbances. The PQS closely approximates the MOS, with a correlation coefficient of more than 0.92, as compared to 0.57 obtained using the conventional weighted mean-square error (WMSE). Extensions and applications of the methodology and of the resulting metric are discussed.

Finite context models improve the performance of chain based encoders to the point that they become attractive, alternative models for binary image compression. The resulting code is within 4% of JBIG at 200 dpi and is 9% more efficient at 400 dpi.

This paper considers an objective picture quality scale for color images (PQScolor). PQScolor approximates the mean opinion score satisfactorily, it takes into account the color perception by using color difference and the properties of visual perception for global features and for localized disturbances. There are two type systems, PQScolor1 is based on Godlove's color difference and PQScolor2 is based on H, V, C signal difference. The correlation coefficient between PQScolor (1, 2) and MOS is more than 0.9, which is very high compared to the value 0.34 obtained for the mean color difference scale.

The design of a wavelet image coder can be divided into three parts: wavelet representation, quantization, and error-free encoding. We evaluate each of these parts individually and synthesize them into complete coders. The evaluation is in the rate-distortion sense; two image quality metrics are used: a perception-based, quantitative picture quality scale (PQS) and the conventional distortion measure, peak signal-to-noise ratio (PSNR). Two representative wavelets, three quantizers, three encoders, and some combinations of these parts are comparatively evaluated. Our results provide an insight into the design issues of optimizing wavelet coders, as well as a good reference for application developers to choose from an increasingly large family of wavelet coders for their applications.

The design of linear image filters based on properties of human visual perception has been shown to require the minimization of criterion functions in both the spatial and frequency domains. We extend this approach to continuous filters of infinite support. For lowpass filters, this leads to the concept of an ideal lowpass image filter that provides a response that is superior perceptually to that of the classical ideal lowpass filter.

An approach to modeling and capturing the time-varying structure of the spectral envelope of speech is reported. Acoustic subword decomposition and the Karhunen-Loeve transform (KLT) are used to extract and efficiently represent the highly correlated structure of the spectral envelope. Integration of the KLT with acoustic subword modeling provides concise representation of both steady-state and dynamic features of the spectra in a unified framework that very effectively captures acoustic-phonetic patterns. The physiological and perceptual basis for the approach, the frame-based and acoustic-subword-based spectral representation, and applications to speaker-dependent recognition are presented. The performance of the recognition algorithm based on this approach compares favorably with that of other techniques.>

For Part I see ibid., vol.1, no.2, p.180-95 (1993). In Part I of this paper, the authors introduced an approach to the representation of the speech spectral envelope which makes use of the Karhunen-Loeve (KL) transformation of acoustic subword segments. This signal-dependent representation captures, with a few KL vectors and transform coefficients, the perceptually and phonetically important structure of the spectral envelope. Here the authors apply this representation to the analysis, synthesis, and coding of speech. They propose simple quantization and coding strategies for the KL representation vectors as well as for the resulting transform coefficients. The resulting technique is a variable rate encoding scheme which achieves good speech quality at an average rate of 3.5 kb/s.>

The authors describe a human visual perception approach and report some results for this problem. The approach is based on the differential quantization of images, in which smooth approximations are subtracted from the image prior to quantization. They consider two such approximations. The first one is an approximation by splines obtained from a sparse and fixed subsampled array of the image. The second one segments the image into piecewise constant regions on the basis of the local activity of the image. Both these approximations result in remainders of residual images where large errors are localized in portions of the image of high activity. Because of visual masking the remainder image can now be coarsely quantized without visual impairment to the reconstructed image. The coarsely quantized remainder is now encoded in an error free manner. In such a perceptually based encoding method the mean square error is now dependent on the activity of the image.>

The authors have developed a signal-dependent representation which captures, with a few KL vectors and transform coefficients, the perceptually and phonetically important structure of the spectral envelope. Together with a mixed excitation strategy with some novel features, this representation has been applied to the analysis, synthesis and coding of speech with promising results in the 5-kb/s range.>

The relationship between the skeleton and the Voronoi diagram for the case of morphologically regular sets is described. Using the discrete-point Voronoi diagram, a practical algorithm is developed to compute an approximate skeleton of a continuous shape from sampled data. The analysis also yields a sampling criterion under which the approximation is stable. The algorithm is applied to a simple test image.>

The goal of this research is to develop interpolation techniques which preserve or enhance the local structure critical to image quality. Preliminary results are presented which exploit either the properties of vision or the properties of the image in order to achieve the goals. Directional image interpolation is considered which is based on a local analysis of the spatial image structure. The extension of techniques for the design of linear filters based on properties of human perception reported previously to enhance the perceived quality of interpolated images is considered.>

A novel approach for speech signal analysis has been developed that incorporates both steady-state and dynamic spectral features into a unified model. This model has been successfully applied in automatic speech recognition contexts and does not require frame-based optimal search algorithms. The model decomposes an utterance into a chain of acoustic subwords and simultaneously generates a mathematical description of instantaneous acoustic-phonetic features and dynamic transitions. The algorithm was tested using a speaker-dependent limited vocabulary recognition task and achieved higher recognition rates than both vector quantization and hidden Markov models.>

Some preliminary results are presented on a simple progressive code for scanned 300 dpi documents. The effectiveness of the standard CCITT codes as a function of document resolution is determined. Extensions to the technique of document resolution are examined. For higher-resolution scanning, e.g. 300 dpi, a pyramid of the information source which allows for a tradeoff between the compression achieved and the quality of the representation is proposed. A progressive code that first generates a color block image that may be useful for identification of the document is proposed. Progression in quality corresponds to 100 dpi binary images, 100 dpi gray scale images, 300 dpi binary images, and finally 300 dpi gray-scale images. Each stage in the progressive code is based on the data available at the previous stage. The performance of the progressive code is examined.>

Algorithms for preprocessing, global modeling and segmentation, local modeling and representation, and binary encoding of facsimile images are proposed, and their performance is examined. Some binary morphological operations that can be used to improve the quality and increase the compressibility of binary images and the modeling of binary images are discussed. The techniques are incorporated into a set of comprehensive encoding schemes, the PCSE codes. Their effectiveness in compressing the International Telegraphy and Telephony Consultative Committee (CCITT) binary images is demonstrated. The PCSE code using preprocessing and color shrinking prior to encoding is shown to outperform the READ code by from 7 to 42% for the standard CCITT test images with a very small change in image quality.>

The authors propose a novel approach to the modeling and estimation of the speech spectral envelope over acoustic subwords that exhibits robust performance in noise. The technique exploits the underlying signal structure of speech to improve parameter estimates, and it uses the perceptual properties of hearing to decrease the computational requirements in a perceptually meaningful way. The approach provides a considerable speech quality improvement over other methods.>

The application of novel anisotropic filter design techniques based on properties of human vision to the processing of luminance and chrominance components of color images is considered. Applied independently, these anisotropic filters can be used for the sequential digital representation of images by subsampling. By using them with two-dimensional quadrature modulation of chrominance signals, they led to a novel scheme for color composite images in which the skewing of energy due to the anisotropy of the filters improves the juxtaposition of luminance and chrominance in the two-dimensional frequency domain. It is found that the image quality is substantially better than that of NTSC images for either sequential or composite techniques.>

A mathematical model has been developed for tracking spectral transitions within the spectral envelope of a speech signal. This technique incorporates linguistic knowledge into a mathematical framework to determine time-varying acoustic-phonetic features and describe formant transitions. The proposed model is quite robust and is capable of extracting not only rapid spectral movement, but also smoother spectral transitions that occur in vowel and sonorant sequences. This basic approach has been previously used to extract steady-state acoustic-phonetic features across spectrally homogeneous regions and to perform speaker dependent recognition in which quite successful results were attained in clean as well as noisy speech. It has now been augmented to capture the dynamics of spectral acoustic-phonetic features.>

The authors present a computationally efficient encoding scheme for vector quantization. Efficiency is achieved by combining techniques: homes are bucketed into the subset of codewords in the same region as the input point; the energy of the input point eliminates codewords not in the same energy range; the smallest hyperrectangle parallel to the coordinate axes that bounds the Voronoi region associated with the codeword acts as a discriminant; and approximations to the actual distortion are used to avoid multiplications. Simulations on Gaussian sources with ranges of codebook sizes and block sizes indicate that the encoding time, measured in multiplications, actually falls with increasing codebook size. It is shown that with no increase in signal/noise ratio the algorithm substantially outperforms tree search and binary hyperplane testing search.>

A full description of image edges requires a complete characterization of their local intensity transitions, the spatial structure of those transitions, and a description of adjacent image regions. The authors propose, as a step toward this end, a 1-D algorithm for describing local intensity transitions by their Gaussian derivatives at a resolution where the support of the Gaussian smoothing matches their widths (blur). The algorithm estimates the transition width from the second derivative of 1-D Gaussian response zero-crossing slope and leads to a characterization of the transition with its first three derivatives at the resolution matching the width. The authors describe how this algorithm can be applied to images and give an example.>

The authors have developed a very successful new approach to automatic speech recognition which incorporates speech knowledge into a mathematical framework and does not require a computationally intensive time alignment/dynamic programming scheme. They transform the speech signal into the spectral domain, segment it into sub-word units and, in turn, perform an additional transformation in the spectral domain to capture the spectral structure within each sub-word unit. The system was shown to perform robustly in hand segmented whole word digit recognition in clean as well as noisy speech. They have now augmented the system with an automatic acoustic sub-word segmentation routine and tested the performance of this integrated system with the TI isolated word database and the confusable E set.>

Advances in computer networking have opened opportunities for remote image processing, allowing processing tasks to be distributed among specialized nodes. The integrated programming environment, Davis interactive system (DAISY) provides a control panel interface on a workstation to a library of image processing applications that run on specialized remote hosts. The workstation provides an interface to a remote image display, image previews, and an image display.>

This paper describes a technique for the enhancement of images by FIR filters which compensates for the decreased response of human vision at high spatial frequencies. Because many images contain mainly horizontal and vertical features, and because vision is less acute along diagonals, it is possible to design anisotropic enhancement filters which do not increase greatly the background noise. The anisotropic design methodology also incorporates other results on filter design based on human vision which have recently been reported by the author and coworkers.

Computer speech recognition is a discipline that has been viewed from two diametrically opposed perspectives. One perspective perceives recognition as a purely mathematical process; the other perceives it as an extensive linguistical "knowledge" base. Because each perspective has its own set of limitations, neither approach has been able to achieve a viable machine realization of human auditory capabilities. Mathematical approaches do not perform fine phonetic distinctions well; linguistical approaches are not suitably machine oriented. We, therefore, propose in this paper a hybrid approach, suited both for machine implementation and for perceiving subtle differences in phonetic structure.

We propose a new, robust pitch detection algorithm for speech degraded by additive noise. Our algorithm exploits the high correlation between adjacent pitch periods that does not exist for the segment as a whole and performs well in the vicinity of voiced/unvoiced regions where the local SNR is low. The algorithm works as follows. We first determine an estimate of the pitch period near the short time peak energy where the local SNR is highest. We then adaptively estimate the local pitch period from the peak towards the transition region by using pitch synchronous cross-correlation with an updated waveform. The performance of this new algorithm is compared to the SIFT and CEPSTRUM algorithms.

The emphasis of many algorithms that have been proposed for the compression of binary images has been the efficient coding of local redundancy in data. We propose that increased compression may be achieved by a decomposition of the compression problem into two steps. The goal of the first step is to extract the global redundancy in an image. This is achieved by a color shrinking algorithm, The goal of the second step is to code the resulting localized data.

Filtering audio signals with filters designed exclusively from frequency domain specifications may result in an audible distortion in the vicinity of sharp amplitude transitions. This paper considers the application of known psychoacoustical properties to the design of digital audio filters which minimizes this distortion while approximating some ideal frequency domain characteristics. Psychoacoustic properties and a simple model for hearing are reviewed. A weighted least squares design criteria based on the model and frequency domain specifications is given. Examples of FIR and IIR filters are given and compared to classical frequency domain filters.

A model based on research in the psychophysics of vision is developed for use in the design of image processing filters in order to quantify the results of image processing as perceived by a human observer. An alternate formulation of the classical frequency-domain filter design problem with both space and frequency-domain specifications is developed. New experimental results concerning the masking effect in the vicinity of edges are reported, and a structure of a model for detection of distortion in complex images is proposed. The ability of the model to predict the visibility of one-dimensional patterns in the vicinity of edges is tested. A distortion measure based on the model is introduced, and practical filter-design criteria are developed. Both FIR linear phase filters and IIR filters are used to demonstrate the applicability of the methods.

For several signal processing applications, the usefulness of Fast Unitary Transforms (FUT) is now well recognized [1-7]. For signal representation, filtering and encoding, it is well known that the Karhunen-Loeve (KL) Transform, based on signal statistics, is optimum in various senses, but the KL Transform is slow. Suboptimum FUT's allow a trade-off between performance and speed. In this paper, we compare and rank the KL, Fourier, Walsh-Hadamard, Haar, Discrete Cosine, Slant Walsh Hadamard and Slant Haar Transforms by their performance in applications and by the number of elementary operations they require. In encoding and filtering, recursive techniques are widely used and are generally fast. By considering both performance and computations we are able to compare directly recursive and transform algorithms. The comparison brings to light a performance versus computation bound for the two classes of processing techniques.

In this paper we derive simple approximate formulas for the performance of entropy-encoded DPCM for a Gaussian random process and a frequency-weighted mean-square distortion measure. Using these results we compare the performance of DPCM to the information theoretic rate-distortion bound. We study the effect on the performance of DPCM of the spectrum of the input process, the frequency weight in the distortion measure, and the number of prediction coefficients. We also examine briefly the case of achromatic still images using line-by-line and two-dimensional DPCM encoding with intrafield and intraframe information.

A set of recursive rules which generate unitary transforms with a fast algorithm (FUT) are presented. For each rule, simple relations give the number of elementary operations required by the fast algorithm. The common Fourier, Walsh-Hadamard (W-H), Haar, and Slant transforms are expressed with these rules. The framework developed allows the introduction of generalized transforms which include all common transforms in a large class of “identical computation transforms”. A systematic and unified view is provided for unitary transforms which have appeared in the literature. This approach leads to a number of new transforms of potential interest. Generalization to complex and multidimensional unitary transforms is considered and some structural relations between transforms are established.

A unified matrix treatment is presented for the various orderings of the Walsh-Hadamard (WH) functions using a general framework. This approach clarifies the different definitions of the WH matrix, the various fast algorithms and the reorderings of the WH functions.

In this correspondence, we comment on some of the shortcomings, for source encoding problems, of the measure of distance between orthonormal representations proposed by Pearl.^1We further examine, using rate-distortion theory, whether rates achievable for a given distortion level by the use of basis-restricted transformations have some inherent dissimilarity independently of input statistics and of other implied parameters.

Most methods of encoding images require complicated implementation. Thus it is of interest to compare the transmission rates that can be achieved by classes of encoding methods of different complexity. We consider two classes of encoding operations. The first class allows any possible operation on the two-dimensional image source output. The second class allows only certain restricted operations on the image. In acquisition of images by electronic means, the image intensity is in general scanned line by line, resulting in data that appear as a sequence of ordinary time series. In encoding, a simpler implementation results if one accepts the time series from a single scan line and encodes it independently of adjacent scan lines. This limits storage requirements to a single scan line and limits processing to operations on a one-parameter time series instead of operations on a two-dimensional field. This is the second class of encoding operations that we consider. We choose a distortion measure and discuss the rate-distortion function, which represents the minimum rate required by any encoding method in the first (arbitrary complex encoding). We then derive the minimum rate that can be achieved by any encoder from the second class. These rates are compared for a specific example.