Human activity recognition (HAR) has become a key technology for improving the quality of life for individuals with special needs, such as older adults and children with Down Syndrome (CwDS). This study presents a novel application of HAR systems to detect directive behaviors in parent-child interactions, focusing on parents of CwDS during educational activities. Using video data, we developed two models: a Convolutional Neural Network (CNN) and a hybrid CNN-LSTM model, to recognize subtle cues such as physical proximity and verbal interactions. The CNN3D model achieved over 90% accuracy in detecting approach behaviors and around 65% for verbal expressions. The CNN-LSTM model outperformed CNN3D in classifying verbal expressions, achieving over 68% accuracy. These results highlight the potential of deep learning classifiers for analyzing subtle parent-child interactions, offering valuable insights into parent-child dynamics and contributing to the development of assistive tools for studying educational settings.

In this work, we aim at identifying the speaker in interactions between mothers and children with Down syndrome (DS) using audio. We collected audio from a session in which children with DS solved puzzles, and their mothers were by their side. We generated a dataset by manually annotating human speech activity and non-speech. We used machine learning to perform four experiments, including individual and generalized models achieving on average F1-scores of 0.74 (five-class model) and 0.84 (two-class) for individual models and 0.69 (five-class) and 0.82 (two-class) for the generalized models. Our results can be helpful to behavioral researchers, therapists, and those interested in better understanding how mother–child interactions unfold in naturalistic settings.

Context identification is one of the key challenges in Ubicomp. An application example is providing contextual information to caregivers of person with dementia to identify assistance needs. Environmental audio provides significant and representative information of the context and the challenge is to automatically identify audio cues coming from overlapping sound sources without sophisticated microphone arrangements. My thesis proposes a succinct representation of the audio, based on the spectral entropy of the signal, and we show experimentally its robustness to source overlap and noise. This would permit ubiquitous applications that perform sound-based activity identification directly in mobile phones.