This paper describes an electro-mechanical and software architecture for the development of an anthropomorphic 6 DOF robotic jaw. The architecture comprises the aggregate of motion components needed to position a prosthetic jaw in 3D space. This architecture frees the jaw kinematics from the dependency of constraints in the mechanical assembly allowing for the fabrication of mechanical systems that can simulate jaw motions beyond human capabilities. The orthogonal and concurrent nature of the structural design makes systems based on this idea potentially the easiest to control. To illustrate our concept two distinct prototypes are produced. Construction of these models resulted in the first two modular anthropomorphic robotic jaws to be built with 6-DOF. Areas of application for this mechanical design include dentistry, speech research, and facial gesture affect research.

A method is described for video-based tracking and analysis of talking heads. Although dots on the face and head are tracked now, clean faces will be tracked once a satisfactory optical flow technique is found. The analysis consists of decomposing the head and face motion components into a common mode and the residual, thus eliminating the need for complex marker tracking and head correction of face motion.

We argue here that examination of the stimulus source is a prerequisite to understanding how audiovisual (AV) stimuli are processed perceptually. This is based on mounting evidence that the act of speech production generates multimodal events whose audible and visible components are highly correlated with each other and the vocal tract source. How this multimodal structuring is exploited perceptually, or not, needs to be demonstrated by conducting studies that take the properties of the stimulus source into account.

The speech acoustics and the phonetically relevant motion of the face during speech are determined by the time-varying behavior of the vocal tract. A benefit of this linkage is that we are able to estimate face motion from the spectral acoustics during speech production using simple neural networks. Thus far, however, the scope of reliable estimation has been limited to individual sentences; network training degrades sharply when multiple sentences are analyzed together. While there is a number of potential avenues for improving network generalization, this paper investigates the possibility that the experimental recording procedures introduce artificial boundary constraints between sentence length utterances. Specifically, the same sentence materials were recorded individually and as part of longer, paragraph length utterances. The scope of reliable network estimation was found to depend both on the length of the utterance (sentence versus paragraph) and, not surprisingly, on phonetic content: estimation of face motion from speech acoustics was reliable for larger sentence training sets when sentences were recorded in continuous paragraph readings; and greater phonetic diversity reduced reliability.

This paper presents an Enhanced analysis-by-synthesis (AbS) Waveform Interpolative (EWI) speech coder at 4 kbps. The system incorporates novel features such as: AbS quantization of the slowly evolving waveform (SEW), AbS vector quantization (VQ) of the dispersion phase, a special pitch search for transitions, and switched-predictive analysis-by-synthesis gain VQ. Subjective quality tests indicate that it exceeds MPEG-4 at 4 kbps and of G.723.1 at 5.3 kbps, and it is slightly better than G.723.1 at 6.3 kbps.

This paper examines the degrees of correlation among vocal-tract and facial movement data and the speech acoustics. Multilinear techniques are applied to support the claims that facial motion during speech is largely a by-product of producing the speech acoustics and further that the spectral envelope of the speech acoustics can be better estimated by the 3D motion of the face than by the midsagittal motion of the anterior vocal-tract (lips, tongue and jaw). Experimental data include measurements of the motion of markers placed on the face and in the vocal-tract, as well as the speech acoustics, for two subjects. The numerical results obtained show that, for both subjects, 91% of the total variance observed in the facial motion data could be determined from vocal-tract motion by means of simple linear estimators. For the inverse path, i.e. recovery of vocal-tract motion from facial motion, the results indicate that about 80% of the variance observed in the vocal-tract can be estimated from the face. Regarding the speech acoustics, it is observed that, in spite of the nonlinear relation between vocal-tract geometry and acoustics, linear estimators are sufficient to determine between 72 and 85% (depending on subject and utterance) of the variance observed in the RMS amplitude and LSP parametric representation of the spectral envelope. A dimensionality analysis is also carried out, and shows that between four and eight components are sufficient to represent the mappings examined. Finally, it is shown that even the tongue, which is an articulator not necessarily coupled with the face, can be recovered reasonably well from facial motion since it frequently displays the same kind of temporal pattern as the jaw during speech.

This paper examines the influence of head orientation in liptracking. There are two main conclusions: First, lip gesture analysis and head movement correction should be processed independently. Second, the measurement of articulatory parameters may be corrupted by head movement if it is performed directly at the pixel level. We thus propose an innovative technique of liptracking which relies on a active contour model of the lips controlled by articulatory parameters. The 3D model is projected onto the image of a speaking face through a camera model, thus allowing spatial re-orientation of the head. Liptracking is then performed by automatic adjustment of the control parameters, independently of head orientation. The final objective of our study is to apply a pixel-based method to detect head orientation. Nevertheless, we consider that head motion and lip gestures are detected by different processes, whether cognitive (by humans) or computational (by machines). Due to this, we decided to first develop and evaluate orientation-free liptracking through a non video-based head motion detection technique which is here presented.

In this paper, vocal tract and orofacial motions are measured during speech production in order to demonstrate that vocal tract motion can be used to estimate its orofacial counterpart. The inversion, i.e. vocal tract behavior estimation from orofacial motion, is also possible, but to a smaller extent. The numerical results showed that vocal tract motion accounted for 96% of the total variance observed in the joint system, whereas orofacial motion accounted for 77%. This analysis is part of a wider study where a dynamical model is being developed to express vocal tract and orofacial motions as a function of muscle activity. This model, currently implemented through multilinear second order autoregressive techniques is described briefly. Finally, the strong direct influence that vocal tract and facial motions have on the energy of the speech acoustics is exemplified. 1 INTRODUCTION For some time now, our goal has been to model speech production using computed mappings between observed ...