We describe an interactive 3D system to provide talkers with real-time information concerning their tongue and jaw movements during speech. Speech movement is tracked by a magnetometer system (Wave; NDI, Waterloo, Ontario, Canada). A customized interface allows users to view their current tongue position (represented as an avatar consisting of flesh-point markers and a modeled surface) placed in a synchronously moving, transparent head. Subjects receive augmented visual feedback when tongue sensors achieve the correct place of articulation. Preliminary data obtained for a group of adult talkers suggest this system can be used to reliably provide real-time feedback for American English consonant place of articulation targets. Future studies, including tests with communication disordered subjects, are described.

Speaking requires the coordinated movements of individual articulators. Understanding each articulator’s contribution to speech is fundamental not only for understanding how speech is produced, but also for optimizing speech assessment and treatment. Our recent work has studied the individual contributions of tongue tip, tongue blade, tongue body front, tongue body back, upper lip, and lower lip movement to speech sound production by tracking the motion of sensors attached on the midline of tongue and lips. An optimal set of articulators (tongue tip, tongue body back, upper lip, and lower lip) has been found. However, the tongue lateral (side)’s contribution to speech is still poorly understood. We therefore investigated the contribution of the tongue lateral region to consonant production by analyzing the motion of a sensor attached to the side of tongue. Repeated productions of 12 consonants (including the lateral approximant /l/) were collected from six native English speakers. Consonant classification accuracy based on articulatory movement data obtained using a support vector machine was used as an indication of contribution level. The results suggest that sagittal movement of the tongue lateral sensor did not significantly benefit consonant classification, over and above the optimal set. Implications of these findings are discussed.

In this paper, a physics-based framework is presented to visualize the human tongue deformation. The tongue is modeled with the Finite Element Method (FEM) and driven by the motion capture data gathered during speech production. Several novel deformation visualization techniques are presented for in-depth data analysis and exploration. To reveal the hidden semantic information of the tongue deformation, we present a novel physics-based volume segmentation algorithm. This is accomplished by decomposing the tongue model into segments based on its deformation pattern with the computation of deformation subspaces and fitting the target deformation locally at each segment. In addition, the strain energy is utilized to provide an intuitive low-dimensional visualization for the high-dimensional sequential motion. Energy-interpolation-based morphing is also equipped to effectively highlight the subtle differences of the 3D deformed shapes without any visual occlusion. Our experimental results and analysis demonstrate the effectiveness of this framework. The proposed methods, though originally designed for the exploration of the tongue deformation, are also valid for general deformation analysis of other shapes.