Understanding the communicative behaviors of non- and minimally-speaking individuals with autism spectrum disorder (ASD) and complex neurodevelopmental disorders (NDDs) remains a critical challenge for both clinical support and machine learning (ML) research. However, developing automated systems for this task is hindered by data scarcity, privacy concerns, heterogeneous and idiosyncratic behaviors, and the significant domain shift from neurotypical to neurodiverse populations. To address these challenges, we first present a novel, large-scale, privacy-preserving action recognition dataset with 2,721 3D skeleton samples capturing in-home interactions of individuals with ASD and complex NDDs. Second, we propose AXON, a novel cross-modal knowledge distillation method that transfers the rich semantic understanding of a pre-trained CLIP model to a graph-based Hyperformer model, outperforming other cross-modal knowledge distillation baselines in action recognition. We further introduce a gradient-based interpretability method to characterize how individuals with ASD and complex NDDs perform communicative actions. Our analysis uncovers both individual- and population-level communicative styles, tendencies, and biases. Our foundational study helps spur the development of more adaptive and personalized augmentative technologies, aiming to foster greater communicative autonomy and understanding for this underserved population.

Nonverbal vocalizations are an essential component of human communication, conveying rich information without linguistic content.However, their computational analysis is hindered by a lack of lexical anchors in the data, compounded by biased and imbalanced data distributions.While disentangled representation learning has shown promise in isolating specific speech features, its application to nonverbal vocalizations remains unexplored.In this paper, we introduce N-CORE, a novel backbone-agnostic framework designed to disentangle intertwined features like emotion and speaker information from nonverbal vocalizations by leveraging N views of audio samples to learn invariance to specific transformations.N-CORE achieves competitive performance compared to state-of-the-art methods for emotion and speaker classification on the VIVAE, ReCANVo, and ReCANVo-Balanced datasets.We further propose an emotion perturbation function that disrupts affective information while preserving speaker information in audio signals for emotion-invariant speaker classification.Our work informs research directions on paralinguistic speech processing, including clinical diagnoses of atypical speech and longitudinal analysis of communicative development.

Nonverbal vocalizations serve an essential role in general communication and expression, and they are particularly integral for non- and minimally-speaking individuals with autism and other neurodevelopmental disorders. Accurate analysis and classification of these vocalizations are crucial for enhancing communication and deepening our understanding of this underserved population. However, most audio-processing machine-learning techniques rely on linguistic cues like speech fluency and word pronunciation, and they do not generalize well to nonverbal vocalizations. We propose a multi-level fusion network that combines three types of audio features – Wav2Vec2 representations, mel-spectrograms, and low-level descriptors – using 6,551 samples across 7 vocalization classes from the open-access ReCANVo dataset. The model achieves an accuracy of 64.09% for the 7-class classification task, outperforming 8 traditional audio classification methods and 3 feature-fusion approaches. To address heterogeneity and noise in the real-world audio data, we tested four sample augmentation techniques, resulting in a 14% relative increase in accuracy. We further examined a single individual (1,595 samples), obtaining a 6.29% relative increase in accuracy, highlighting the effects of speaker and environment variability in population-level models. We also achieved a binary classification accuracy of 85.80% for vocalizations associated with positive/negative affective states, suggesting a potentially robust and highly separable latent structure underlying the valence of these sounds. Finally, we quantified the inter-class relationships between vocalizations using cosine similarity to offer additional insights into the acoustic patterns between classes. Our exploratory study not only uncovers key challenges in processing nonverbal vocalizations but also provides a foundational framework for future machine learning research with rare, real-world audio data, including multi-feature fusion and sample augmentation.

Mental health concerns are a critical issue in today’s digital age, posing a threat to both individual and societal well-being and making the identification of at-risk individuals crucial. Analyzing an individual’s social media post history can offer insights into their mental health state and help identify the presence of suicidal ideation. However, the complexity of linguistic and temporal data, along with sparsity and time irregularities, poses a formidable challenge in machine learning. Previous methods in this domain either rely on Euclidean space for processing which does not adequately model the power-law properties of social media posts, or lose information due to the discretization of the time axis. To address these challenges, we propose a novel framework, THYMES, which leverages pre-trained encoders and a rich representation learning paradigm with hyperbolic learning to model power-law features for enhanced sequence modeling. We perform experiments on two datasets and demonstrate that THYMES outperforms previously proposed methods while maintaining classification fairness under heavy data imbalances. Additionally, we qualitatively analyze commonly misclassified samples to reveal the shortcomings of models in this domain.

The rising rate of mental health issues in the digital age underscores the critical need for proactive interventions to assess an individual’s well-being. This problem is further exacerbated by the social stigma surrounding the subject, which suppresses the willingness of victims to seek help. Social media can serve as an outlet for such individuals to express their negative emotions or thoughts of self-harm. The social media account of an individual can offer a plethora of valuable information that can be used to predict their mental health. By unifying principles of robust classifier training and selective classification, we propose a novel framework, SAFENet, to predict the suicide risk of users by using their historical social media posts. When the confidence of prediction is low or the individual is classified as a high-risk user, SAFENet delegates the analysis of the posts to a human evaluator for further intervention. Our experiments show that SAFENet outperforms existing state-of-the-art frameworks. We further qualitatively analyze predictions from SAFENet and demonstrate that it performs robustly on difficult samples that may cause contemporary methods to make errors. Our system addresses the urgent need for efficient and effective mental health intervention in the digital era.

The complexity of text-embedded images presents a formidable challenge in machine learning given the need for multimodal understanding of multiple aspects of expression conveyed by them.While previous research in multimodal analysis has primarily focused on singular aspects such as hate speech and its subclasses, this study expands this focus to encompass multiple aspects of linguistics: hate, targets of hate, stance, and humor.We introduce a novel dataset PrideMM comprising 5,063 text-embedded images associated with the LGBTQ+ Pride movement, thereby addressing a serious gap in existing resources.We conduct extensive experimentation on PrideMM by using unimodal and multimodal baseline methods to establish benchmarks for each task.Additionally, we propose a novel framework MemeCLIP for efficient downstream learning while preserving the knowledge of the pre-trained CLIP model.The results of our experiments show that Meme-CLIP achieves superior performance compared to previously proposed frameworks on two real-world datasets.We further compare the performance of MemeCLIP and zero-shot GPT-4 on the hate classification task.Finally, we discuss the shortcomings of our model by qualitatively analyzing misclassified samples.