This paper presents specific new techniques for coding of speech representations and a new general approach to coding for compression that directly utilizes the multidimensional nature of the input data. Many methods of speech analysis yield a two-dimensional (2-D) pattern, with time as one of the dimensions. Various such speech representations, and power spectrum sequences in particular, are shown here to be amenable to 2-D compression using specific models which take account of a large part of their structure in both dimensions. Newly developed techniques, multistep adaptive flux interpolation (MAFI) and multistep flow-based prediction (MFBP) are presented. These are able to code power spectral density (PSD) sequences of speech more completely and accurately than conventional methods. This is due to their ability to model nonstationary, but piecewise-continuous, signals, of which speech is a good example. Initially, MAFI and MFBP are applied in the time domain, then reapplied to the encoded data in the second dimension. This approach allows the coding algorithm to exploit redundancy in both dimensions, giving a significant improvement in the overall compression ratio. Furthermore, the compression may be reapplied several times. The data is further compressed with each application.

A two-dimensional discrete cosine transform (2-D DCT), often used for image coding, has been applied to sequences of speech spectra produced by the maximum likelihood method (MLM). The coded data was compressed by nearly 90%, reducing it to a size smaller than that needed to store the coefficients of a 10th order linear predictive coding (LPC) model. The DCT-encoded data was then reconstructed and tested for intelligibility. It was found that the two-dimensional DCT method was significantly more intelligible and more natural-sounding than the LPC technique.

This paper describes a novel approach for visual speech recognition. The shape of the mouth is modelled by an active shape model which is derived from the statistics of a training set and used to locate, track and parameterise the speaker's lip movements. The extracted parameters representing the lip shape are modelled as continuous probability distributions and their temporal dependencies are modelled by hidden Markov models. We present recognition tests performed on a database of a broad variety of speakers and illumination conditions. The system achieved an accuracy of 85.42% for a speaker independent recognition task of the first four digits using lip shape information only.

This paper describes a robust method for extracting visual speech information from the shape of lips to be used for an automatic speechreading (lipreading) systems. Lip deformation is modelled by a statistically based deformable contour model which learns typical lip deformation from a training set. The main difficulty in locating and tracking lips consists of finding dominant image features for representing the lip contours. We describe the use of a statistical profile model which learns dominant image features from a training set. The model captures global intensity variation due to different illumination and different skin reflectance as well as intensity changes at the inner lip contour due to mouth opening and visibility of teeth and tongue. The method is validated for locating and tracking lip movements on a database of a broad variety of speakers.

An approach for person identification is described based on spatio-temporal analysis of the talking face. A person is represented by a parametric model of the visible speech articulators and their temporal characteristics during speech production. The model consists of shape parameters, representing the lip contour and intensity parameters representing the grey level distribution in the mouth region. The model is used to track lips in image sequences where the model parameters are recovered from the tracking results. While some of these parameters relate to speech information, others are intuitively related to different persons and we show that models based on these features enable successful person identification. We model the shape and intensity parameters as mixtures of Gaussians and their temporal dependencies by hidden Markov models. Identifying a talking person is performed by estimating the likelihood of each model for having generated the observed sequence of features and the model with the highest likelihood is chosen as the identified person.

We describe a speechreading system that uses both, shape information from the lip contours and intensity information from the mouth area. Shape information is obtained by tracking and parameterising the inner and outer lip boundary in an image sequence. Intensity information is extracted from a grey level model, based on principal component analysis. In comparison to other approaches, the intensity area deforms with the shape model to ensure that similar object features are represented after non-rigid deformation of the lips. We describe speaker independent recognition experiments based on these features and Hidden Markov Models. Preliminary results suggest that similar performance can be achieved by using either shape or intensity information and slightly higher performance by their combined use.

This paper describes a new approach for speaker identification based on lipreading. Visual features are extracted from image sequences of the talking face and consist of shape parameters which describe the lip boundary and intensity parameters which describe the grey-level distribution of the mouth area. Intensity information is based on principal component analysis using eigenspaces which deform with the shape model. The extracted parameters account for both, speech dependent and speaker dependent information. We built spatio-temporal speaker models based on these features, using HMMs with mixtures of Gaussians. Promising results were obtained for text dependent and text independent speaker identification tests performed on a small video database.

This paper introduces a model being developed for estimating child and adolescent formant frequency values from adult data.The model approximates adult male and female pharyngeal and oral cavity lengths, and scales these along the corresponding male or female growth curve.The second and third formant frequencies are estimated directly from these scaled vocal tract dimensions.Two methods of establishing the first formant from the scaled data are discussed.Initial results obtained in the scaling of adult data to child values suggest that age, height and gender are all significant when estimating child formant frequency values.Furthermore, averaging of male and female data is found to be inappropriate since the differing growth rates of males and females imply that vocal tract dimensions cannot be linearly related.

The information produced by a straightforward auditory model, like the original speech signal, is subject to extreme local variability and is not suitable for direct presentation to speech recognition algorithms. Therefore, an analysis is required which will make the important features of the neural firing patterns explicit, but which will describe them concisely and without disruption by known characteristics of speech. Several attempts have been made to develop such an analysis, however, such analyses ignore significant aspects of neural firing patterns, and consequently information is lost (usually that associated with fine temporal structure). The authors describe a 'reduced auditory representation', which retains more information, and presents the results of preliminary recognition experiments using this data.>