Dysarthria is a motor speech disorder often characterized by reduced speech intelligibility through slow, uncoordinated control of speech production muscles. Automatic Speech recognition (ASR) systems may help dysarthric talkers communicate more effectively. To have robust dysarthria-specific ASR, sufficient training speech is required, which is not readily available. Recent advances in Text-To-Speech (TTS) synthesis multi-speaker end-to-end systems suggest the possibility of using synthesis for data augmentation. In this paper, we aim to improve multi-speaker end-to-end TTS systems to synthesize dysarthric speech for improved training of a dysarthria-specific DNN-HMM ASR. In the synthesized speech, we add dysarthria severity level and pause insertion mechanisms to other control parameters such as pitch, energy, and duration. Results show that a DNN-HMM model trained on additional synthetic dysarthric speech achieves WER improvement of 12.2% compared to the baseline, the addition of the severity level and pause insertion controls decrease WER by 6.5%, showing the effectiveness of adding these parameters. Audio samples are available at https://mohammadelc.github.io/SpeechGroupUKY/

Acoustic-to-articulatory inversion, the estimation of articulatory kinematics from an acoustic waveform, is a challenging but important problem. Accurate estimation of articulatory movements has the potential for significant impact on our understanding of speech production, on our capacity to assess and treat pathologies in a clinical setting, and on speech technologies such as computer aided pronunciation assessment and audio-video synthesis. However, because of the complex and speaker-specific relationship between articulation and acoustics, existing approaches for inversion do not generalize well across speakers. As acquiring speaker-specific kinematic data for training is not feasible in many practical applications, this remains an important and open problem. This paper proposes a novel approach to acoustic-to-articulatory inversion, Parallel Reference Speaker Weighting (PRSW), which requires no kinematic data for the target speaker and a small amount of acoustic adaptation data. PRSW hypothesizes that acoustic and kinematic similarities are correlated and uses speaker-adapted articulatory models derived from acoustically derived weights. The system was assessed using a 20-speaker data set of synchronous acoustic and Electromagnetic Articulography (EMA) kinematic data. Results demonstrate that by restricting the reference group to a subset consisting of speakers with strong individual speaker-dependent inversion performance, the PRSW method is able to attain kinematic-independent acoustic-to-articulatory inversion performance nearly matching that of the speaker-dependent model, with an average correlation of 0.62 versus 0.63. This indicates that given a sufficiently complete and appropriately selected reference speaker set for adaptation, it is possible to create effective articulatory models without kinematic training data.

Articulatory data such as that collected via electromagnetic articulography (EMA) are valuable for many speech applications, including speech modeling, recognition, and synthesis. Nearly all current EMA applications and methods focus on the use of positional sensor data, even though modern 3D-EMA systems also capture sensor orientation, which provides significant additional information about articulator posture and vocal tract shape. To address this problem, this paper introduces a new method for interpolating untracked tongue fleshpoint positions from the combined position and orientation data of three EMA sensors on the tongue, using additional reference sensors for evaluating interpolation error. Comparison of interpolated and measured data illustrated effectiveness of the new method in providing additional tongue shape and position features. The results suggest that analytic methods that combine sensor position and orientation data are able to improve the characterization of tongue kinematics even using a small number of EMA sensors.

Sensorimotor adaptation is an important focus in the study of motor learning for non-disordered speech, but has yet to be studied substantially for speech rehabilitation. Speech adaptation is typically elicited experimentally using LPC resynthesis to modify the sounds that a speaker hears himself producing. This method requires that the participant be able to produce a robust speech-acoustic signal and is therefore not well-suited for talkers with dysarthria. We have developed a novel technique using electromagnetic articulography (EMA) to drive an articulatory synthesizer. The acoustic output of the articulatory synthesizer can be perturbed experimentally to study auditory feedback effects on sensorimotor learning. This work aims to compare sensorimotor adaptation effects using our articulatory resynthesis method with effects from an established, acoustic-only method. Results suggest that the articulatory resynthesis method can elicit speech adaptation, but that the articulatory effects of the two methods differ.

There is a significant need for more comprehensive electromagnetic articulography (EMA) datasets that can provide matched acoustics and articulatory kinematic data with good spatial and temporal resolution. The Marquette University Electromagnetic Articulography Mandarin Accented English (EMA-MAE) corpus provides kinematic and acoustic data from 40 gender and dialect balanced speakers representing 20 Midwestern standard American English L1 speakers and 20 Mandarin Accented English (MAE) L2 speakers, half Beijing region dialect and half are Shanghai region dialect. Three dimensional EMA data were collected at a 400 Hz sampling rate using the NDI Wave system, with articulatory sensors on the midsagittal lips, lower incisors, tongue blade and dorsum, plus lateral lip corner and tongue body. Sensors provide three-dimensional position data as well as two-dimensional orientation data representing the orientation of the sensor plane. Data have been corrected for head movement relative to a fixed reference sensor and also adjusted using a biteplate calibration system to place the data in an articulatory working space relative to each subject's individual midsagittal and maxillary occlusal planes. Speech materials include isolated words chosen to focus on specific contrasts between the English and Mandarin languages, as well as sentences and paragraphs for continuous speech, totaling approximately 45 minutes of data per subject. A beta version of the EMA-MAE corpus is now available, and the full corpus is in preparation for public release to help advance research in areas such as pronunciation modeling, acoustic-articulatory inversion, L1-L2 comparisons, pronunciation error detection, and accent modification training.

Speech sensorimotor adaptation is the short-term learning of modified articulator movements evoked through sensory-feedback perturbations. A common experimental method manipulates acoustic parameters, such as formant frequencies, using real time resynthesis of the participant's speech to perturb auditory feedback. While some studies have examined phrases comprised of vowels, diphthongs, and semivowels, the bulk of research on auditory feedback-driven sensorimotor adaptation has focused on vowels in neutral contexts (/hVd/). The current study investigates coarticulatory influences of adjacent consonants on sensorimotor adaptation. The purpose is to evaluate differences in the adaptation effects for vowels in consonant environments that vary by place and manner of articulation. In particular, we addressed the hypothesis that contexts with greater intra-articulator coarticulation and more static articulatory postures (alveolars and fricatives) offer greater resistance to vowel adaptation than contexts with primarily inter-articulator coarticulation and more dynamic articulatory patterns (bilabials and stops). Participants completed formant perturbation-driven vowel adaptation experiments for varying CVCs. Results from discrete formant measures at the vowel midpoint were generally consistent with the hypothesis. Analyses of more complete formant trajectories suggest that adaptation can also (or alternatively) influence formant onsets, offsets, and transitions, resulting in complex formant pattern changes that may reflect modifications to consonant articulation

Broadening our understanding of the components and processes of speech sensorimotor learning is crucial to furthering methods of speech neurorehabilitation. Recent research in limb sensorimotor control has used virtual environments to study learning in novel sensorimotor working spaces. Comparable experimental paradigms have yet to be undertaken to study speech learning. We present acoustic and kinematic data obtained from participants producing vowels in unfamiliar articulatory-acoustic working spaces using a virtual vocal tract. Talkers with dysarthria and healthy controls were asked to produce vowels using an electromagnetic articulograph-driven speech synthesizer for participant-controlled auditory feedback. The aim of the work was to characterize performance within and between groups to generate hypotheses regarding experimental manipulations that may bolster our understanding of speech sensorimotor learning. Results indicate that dysarthric talkers displayed relatively reduced acoustic working spaces and somewhat more variable acoustic targets compared to controls. Kinematic measures of articulatory dynamics, particularly peak speed and movement jerk-cost, were idiosyncratic and did not dissociate talker groups. These findings suggest that individuals with dysarthria and healthy talkers may use idiosyncratic movement strategies in learning to control a virtual vocal tract, but that dysarthric talkers may nonetheless exhibit acoustic limitations that parallel deficits in speech intelligibility.

The selection of effective articulatory features is an important component of tasks such as acoustic-to-articulator inversion and articulatory synthesis. Although it is common to use direct articulatory sensor measurements as feature variables, this approach fails to incorporate important physiological information such as palate height and shape and thus is not as representative of vocal tract cross section as desired. We introduce a set of articulator feature variables that are palate referenced and normalized with respect to the articulatory working space in order to improve the quality of the vocal tract representation. These features include normalized horizontal positions plus the normalized palatal height of two midsagittal and one lateral tongue sensor, as well as normalized lip separation and lip protrusion. The quality of the feature representation is evaluated subjectively by comparing the variances and vowel separation in the working space and quantitatively through measurement of acoustic-to-articulator inversion error. Results indicate that the palate-referenced features have reduced variance and increased separation between vowels spaces and substantially lower inversion error than direct sensor measures.