Understanding the communicative behaviors of non- and minimally-speaking individuals with autism spectrum disorder (ASD) and complex neurodevelopmental disorders (NDDs) remains a critical challenge for both clinical support and machine learning (ML) research. However, developing automated systems for this task is hindered by data scarcity, privacy concerns, heterogeneous and idiosyncratic behaviors, and the significant domain shift from neurotypical to neurodiverse populations. To address these challenges, we first present a novel, large-scale, privacy-preserving action recognition dataset with 2,721 3D skeleton samples capturing in-home interactions of individuals with ASD and complex NDDs. Second, we propose AXON, a novel cross-modal knowledge distillation method that transfers the rich semantic understanding of a pre-trained CLIP model to a graph-based Hyperformer model, outperforming other cross-modal knowledge distillation baselines in action recognition. We further introduce a gradient-based interpretability method to characterize how individuals with ASD and complex NDDs perform communicative actions. Our analysis uncovers both individual- and population-level communicative styles, tendencies, and biases. Our foundational study helps spur the development of more adaptive and personalized augmentative technologies, aiming to foster greater communicative autonomy and understanding for this underserved population.

Nonverbal vocalizations are an essential component of human communication, conveying rich information without linguistic content.However, their computational analysis is hindered by a lack of lexical anchors in the data, compounded by biased and imbalanced data distributions.While disentangled representation learning has shown promise in isolating specific speech features, its application to nonverbal vocalizations remains unexplored.In this paper, we introduce N-CORE, a novel backbone-agnostic framework designed to disentangle intertwined features like emotion and speaker information from nonverbal vocalizations by leveraging N views of audio samples to learn invariance to specific transformations.N-CORE achieves competitive performance compared to state-of-the-art methods for emotion and speaker classification on the VIVAE, ReCANVo, and ReCANVo-Balanced datasets.We further propose an emotion perturbation function that disrupts affective information while preserving speaker information in audio signals for emotion-invariant speaker classification.Our work informs research directions on paralinguistic speech processing, including clinical diagnoses of atypical speech and longitudinal analysis of communicative development.

Nonverbal vocalizations serve an essential role in general communication and expression, and they are particularly integral for non- and minimally-speaking individuals with autism and other neurodevelopmental disorders. Accurate analysis and classification of these vocalizations are crucial for enhancing communication and deepening our understanding of this underserved population. However, most audio-processing machine-learning techniques rely on linguistic cues like speech fluency and word pronunciation, and they do not generalize well to nonverbal vocalizations. We propose a multi-level fusion network that combines three types of audio features – Wav2Vec2 representations, mel-spectrograms, and low-level descriptors – using 6,551 samples across 7 vocalization classes from the open-access ReCANVo dataset. The model achieves an accuracy of 64.09% for the 7-class classification task, outperforming 8 traditional audio classification methods and 3 feature-fusion approaches. To address heterogeneity and noise in the real-world audio data, we tested four sample augmentation techniques, resulting in a 14% relative increase in accuracy. We further examined a single individual (1,595 samples), obtaining a 6.29% relative increase in accuracy, highlighting the effects of speaker and environment variability in population-level models. We also achieved a binary classification accuracy of 85.80% for vocalizations associated with positive/negative affective states, suggesting a potentially robust and highly separable latent structure underlying the valence of these sounds. Finally, we quantified the inter-class relationships between vocalizations using cosine similarity to offer additional insights into the acoustic patterns between classes. Our exploratory study not only uncovers key challenges in processing nonverbal vocalizations but also provides a foundational framework for future machine learning research with rare, real-world audio data, including multi-feature fusion and sample augmentation.

Individuals who produce few spoken words (“minimally-speaking” individuals) often convey rich affective and communicative information through nonverbal vocalizations, such as grunts, yells, babbles, and monosyllabic expressions. Yet, little data exists on the affective content of the vocal expressions of this population. Here, we present 78,624 arousal and valence ratings of nonverbal vocalizations from the online ReCANVo (Real-World Communicative and Affective Nonverbal Vocalizations) database. This dataset contains over 7,000 vocalizations that have been labeled with their expressive functions (delight, frustration, etc.) from eight minimally-speaking individuals. Our results suggest that raters who have no knowledge of the context or meaning of a nonverbal vocalization are still able to detect arousal and valence differences between different types of vocalizations based on Likert-scale ratings. Moreover, these ratings are consistent with hypothesized arousal and valence rankings for the different vocalization types. Raters are also able to detect arousal and valence differences between different vocalization types within individual speakers. To our knowledge, this is the first large-scale analysis of affective content within nonverbal vocalizations from minimally verbal individuals. These results complement affective computing research of nonverbal vocalizations that occur within typical verbal speech (e.g., grunts, sighs) and serve as a foundation for further understanding of how humans perceive emotions in sounds.

Nonverbal vocalizations from non- and minimally speaking individuals who speak fewer than 20 words (mv* individuals) convey important communicative and affective information. While nonverbal vocalizations that occur amidst typical speech and infant vocalizations have been studied extensively in the literature, there is limited prior work on vocalizations by mv* individuals. Our work is among the first studies of the communicative and affective information expressed in nonverbal vocalizations by mv* children and adults. We collected labeled vocalizations in real-world settings with eight mv* communicators, with communicative and affective labels provided in-the-moment by a close family member. Using evaluation strategies suitable for messy, real-world data, we show that nonverbal vocalizations can be classified by function (with 4- and 5-way classifications) with F1 scores above chance for all participants. We analyze labeling and data collection practices for each participating family, and discuss the classification results in the context of our novel real-world data collection protocol. The presented work includes results from the largest classification experiments with nonverbal vocalizations from mv* communicators to date.

Nonverbal vocalizations contain important affective and communicative information, especially for those who do not use traditional speech, including individuals who have autism and are non- or minimally verbal (nv/mv). Although these vocalizations are often understood by those who know them well, they can be challenging to understand for the community-at-large. This work presents (1) a methodology for collecting spontaneous vocalizations from nv/mv individuals in natural environments, with no researcher present, and personalized in-the-moment labels from a family member; (2) speaker-dependent classification of these real-world sounds for three nv/mv individuals; and (3) an interactive application to translate the nonverbal vocalizations in real time. Using support-vector machine and random forest models, we achieved speaker-dependent unweighted average recalls (UARs) of 0.75, 0.53, and 0.79 for the three individuals, respectively, with each model discriminating between 5 nonverbal vocalization classes. We also present first results for real-time binary classification of positive- and negative-affect nonverbal vocalizations, trained using a commercial wearable microphone and tested in real time using a smartphone. This work informs personalized machine learning methods for non-traditional communicators and advances real-world interactive augmentative technology for an underserved population.

Current research to understand and enhance the development of children with neurological differences, including Autism Spectrum Disorder (ASD), is often severely limited by small sample sizes of human-gathered data in artificially structured learning environments. SPRING: Smart Platform for Research, Intervention, and Neurodevelopmental Growth is a new hardware and software system designed to 1) automate quantitative data acquisition, 2) optimize learning progressions through customized, motivating stimuli, and 3) encourage social, cognitive, and motor development in a personalized, child-led play environment. SPRING can also be paired with sensors to probe the physiological underpinnings of motivation, engagement, and cognition. Here, we present the design principles and methodology for SPRING, as well as two heterogeneous case studies. The first case highlights enhanced attention and accelerated skill development using SPRING, while the second pairs SPRING data with electrodermal activity measurements to identify a possible physiological signature of engagement and challenge in learning.