Gamified biofeedback has the potential to help individuals improve speech motor patterns. For such systems, feedback provided in response to attempted changes must directly characterize performance improvements. This is particularly the case for ultrasound biofeedback therapy (UBT), which is increasingly used to visualize tongue movement for treating misarticulations of speech sounds (e.g., American English /ɹ/). Previous studies on ultrasound imaging of misarticulated /ɹ/ sounds have established that a single, time-dependent measured parameter, δ (relative difference between normalized tongue dorsum and blade displacements) can classify articulatory accuracy, matching human judgments with about 85% agreement. The δ parameter thus is an ideal basis for real-time gamified UBT. However, during ultrasound imaging of a speech production, there are multiple possible image frames from which δ could be automatically evaluated to represent production accuracy. Accordingly, to identify the optimal frame selection for δ evaluation, ultrasound image sequences depicting 2,944 productions of 15 distinct rhotic phrases were analyzed, while perceptual accuracy was judged by trained listeners. Four different selection criteria were compared, with classification performance and optimal thresholds evaluated using receiver-operating-characteristic curve analysis with 8-fold cross validation. For pre-vocalic and post-vocalic rhotic syllables, highest classification success rates resulted from evaluating δ at productions’ end. For pre-vocalic words preceded by the carrier phrase “a”, best classification resulted from evaluating δ at the estimated temporal midpoint of /ɹ/. Results provide a viable basis for gamified ultrasound biofeedback therapy based on real-time ultrasound measurements of tongue movements.

We present a novel Automatic Target Recognition (ATR) system using open-vocabulary object detection and classification models. A primary advantage of this approach is that target classes can be defined just before runtime by a non-technical end user, using either a few natural language text descriptions of the target, or a few image exemplars, or both. Nuances in the desired targets can be expressed in natural language, which is useful for unique targets with little or no training data. We also implemented a novel combination of several techniques to improve performance, such as leveraging the additional information in the sequence of overlapping frames to perform tubelet identification (i.e., sequential bounding box matching), bounding box re-scoring, and tubelet linking. Additionally, we developed a technique to visualize the aggregate output of many overlapping frames as a mosaic of the area scanned during the aerial surveillance or reconnaissance, and a kernel density estimate (or heatmap) of the detected targets. We initially applied this ATR system to the use case of detecting and clearing unexploded ordinance on airfield runways and we are currently extending our research to other real-world applications.

Characterizing accurate vs. misarticulated patterns of tongue movement using ultrasound can be challenging in real time because of the fast, independent movement of tongue regions. The usefulness of ultrasound for biofeedback speech therapy is limited because speakers must mentally track and compare differences between their tongue movement and available models. It is desirable to automate this interpretive task using a single parameter representing deviation from known accurate tongue movements. In this study, displacements recorded automatically by ultrasound image tracking were transformed into a single biofeedback parameter (time-dependent difference between blade and dorsum displacements). Receiver operating characteristic (ROC) curve analysis was used to evaluate this parameter as a predictor of production accuracy over a range of different vowel contexts with initial and final /r/ in American English. Areas under ROC curves were 0.8 or above, indicating that this simple parameter may provide useful real-time biofeedback on /r/ accuracy within a range of rhotic contexts.