This paper presents methods and experimental results for phonetic classification using 39 phone classes and the NIST recommended training and test sets for NTIMIT and TIMIT. Spectral/temporal features which represent the smoothed trajectory of FFT derived speech spectra over 300 ms intervals are used for the analysis. Classification tests are made with both a binary-pair partitioned (BPP) neural network system (one neural network for each of the 741 pairs of phones) and a single large neural network. The classification accuracy is very similar for the two types of networks, but the BPP method has the advantage of a much shorter training time. The best results obtained (77% for TIMIT and 67.4% for NTIMIT) compare favorably to the best results reported in the literature for this task.

Among the various methods that have been proposed to improve the robustness and accuracy of automatic speech recognition (ASR) systems, lipreading has received little attention until very recently. However, results from the psychological literature indicate that lipreading, in conjunction with auditory perception, can provide a strong improvement in speech recognition and understanding in humans. We have developed a novel speaker-dependent lipreading system that uses hidden Markov models. An audiovisual system known as Lipreading to Enhance Automatic Perception of Speech (LEAPS) is described, in which the lipreading system is used in conjunction with an audio ASR system in order to improve the accuracy of the latter, especially under degraded acoustical conditions. Experimental results are presented for two small phoneme discrimination tasks, as well as a medium vocabulary isolated word recognition task. In all cases, performance of the combined system is superior to that of the audio system, with a reduction in errors ranging from 20 to 65%.

Experiments in modeling speech signals for phoneme classification are described. Enhancements to standard speech processing methods include basis vector representations of dynamic feature trajectories, morphological smoothing (dilation) of spectral features, and the use of many closely spaced, short analysis windows. Results are reported from experiments using the TIMIT database of up to 71.0% correct classification of 16 presegmented vowels in a noise-free environment, and 54.5% correct classification in a 10 dB signal-to-noise ratio environment.

The use of motion information in deformable templates for the analysis of image sequences is investigated. Two primary means of utilizing this information are identified. The first is to extend the topological structure of the template into the time dimension. This is accomplished by parameterizing the motion of each point of the template with a set of discrete cosine transform coefficients. The second is to utilize motion information (in this case, simply the pixel-by-pixel frame difference) as a primitive feature. Both methods are demonstrated experimentally, along with a method which combines the two. The first is less susceptible to short-duration anomalous solutions, while the second is able to capture certain features missed by the static analysis. The combined method appears to possess the advantages of both.>

Among the various methods which have been proposed to improve the robustness and accuracy of automatic speech recognition (ASR) systems, lipreading has received very little attention. In this paper, we provide motivation for the use of lipreading. A novel speaker dependent lipreading system is developed, which uses hidden Markov modeling, a well known and highly successful technique for audio-based ASR. It is used in conjunction with an audio ASR system in order to improve the accuracy of the latter, especially under degraded acoustical conditions. Reductions in error of 30 to over 60% result.

Visual pattern image sequence coding (VPISC) is a new digital image sequence (video) coding process that possesses significant advantages relative to other technologies; in particular, it is extremely efficient in terms of the computational effort required. It is designed to exploit properties of the human visual system (HVS), and thus yields high visual fidelity. Visual quality criteria are deliberately chosen over information-theoretic ones on the grounds that, in images intended for human viewing, visual criteria are the most meaningful ones. VPISC yields impressive compression comparable to other recent methods, such as motion-compensated vector quantization. VPISC divides the images into spatiotemporal cubes, which are then independently matched with one of a small, predetermined set of visually meaningful three-dimensional space-time patterns. The pattern set is chosen to conform to specific characteristics of the HVS. Also introduced are two modifications of VPISC: adaptive and foveal VPISC. These are spatiotemporally nonuniform implementations that code different portions of the image sequence at different resolutions, according to either a fidelity criterion (for AVPISC) or a foveation criterion (for FVPISC).>

Hierarchical visual pattern image coding (HVPIC) is a highly successful, low-computational-cost, pyramidal image coding scheme which utilizes human visual system (HVS) properties to achieve very low bit rates while maintaining a good perceived image quality. A description is presented of an extension of HVPIC, termed adaptive VPIC (AVPIC), which results in a decreased bit rate for most images with very little additional computation and virtually no further image degradation. After coding but before transmission, the encoder analyzes what it has done, and reduces the transmission bit rate by eliminating high-resolution information about low-resolution portions of the image. The method is adaptive in the sense that the bit rate is locally adjusted to match the level of detail present in an image region.>

Visual pattern image coding, or VPIC1-5, is an important new digital image coding process that possesses significant advantages relative to all other existing technologies: VPIC is capable of coding (i) with visual fidelity comparable to the best available techniques; (ii) at very high compressions exceeding the best available technologies: compressions in the range 30:1—40:1 are obtained routinely; (iii) with absolutely unprecedented coding efficiency - coding/decoding via VPIC is completely linear with respect to image size and entails a complexity 1-2 orders of magnitude faster than any prior high compression strategy; (iv) configurability. In the current work, the VPIC coding framework developed initially for single images is extended to image sequences. The algorithm for image sequence coding presented here, termed Visual Pattern Image Sequence Coding (or VPISC) exploits all of the advantages of "static" VPIC in the reduction of information from an additional (temporal) dimension to achieve unprecedented image sequence coding performance stated in terms of coding complexity, compression, and visual fidelity.