Public speaking experts intentionally take their audience on an emotional roller coaster, staying attuned to their audience's collective emotional feedback. In this research, we explore how bidirectional sharing of heart rates between a speaker and their audience facilitates this emotional exchange, through empathy, emotional awareness, and engagement. Firstly, in two design studies$(N=25)$, we evaluated this concept and identified design elements for a heart rate sharing interface. Subsequently, we developed Feeling-the-Beat, a system for sharing heart rate between speakers and audiences in real time. Finally, in a randomized, counter-balanced, within-subjects study$(N=36)$, we compared our system to one sharing fabricated heart rates and a baseline system without heart rate sharing. Feeling-the-Beat significantly increased audience empathy towards the speaker, audience engagement during moments of heightened speaker heart rate, and social presence.

It has been suggested that computer-assisted instruction (CAI) is a promising method for educating students on the autism spectrum. We aimed to determine whether automated recognition of facial expressions aided in predicting CAI engagement and learning performance. Seven youth with autism (mean age = 12.7, SD = 4.2) interacted with a CAI program, TeachTown Basics, for 15 consecutive sessions. Video recordings of the participants' faces were collected during these sessions and facial expressions from these videos were analyzed using CERT, an algorithm that automatically outputs intensity values for each facial action unit (AU). Using these data, we attempted to operationally define two engagement indices: (1) behavioral engagement, the proportion of time a participant had their face oriented to the computer screen; and (2) emotional engagement, the activation of AUs previously associated with CAI. Our results suggest that both indices strongly correlated with one another, but that emotional (not behavioral) engagement predicted test performance. CAI knowledge domain, participant sex, and developmental age also contributed to the prediction.

This paper extends previous work automatically detecting stereotypical motor movements (SMM) in individuals on the autism spectrum. Using three-axis accelerometer data obtained through wearable wireless sensors, we compare recognition results for two different classifiers -- Support Vector Machine and Decision Tree -- in combination with different feature sets based on time-frequency characteristics of accelerometer data. We use data collected from six individuals on the autism spectrum who participated in two different studies conducted three years apart in classroom settings, and observe an average accuracy across all participants over time ranging from 81.2% (TPR: 0.91; FPR: 0.21) to 99.1% (TPR: 0.99; FPR: 0.01) for all combinations of classifiers and feature sets. We also provide analyses of kinematic parameters associated with observed movements in an attempt to explain classifier-feature specific performance. Based on our results, we conclude that real-time, person-dependent, adaptive algorithms are needed in order to accurately and consistently measure SMM automatically in individuals on the autism spectrum over time in real-word settings.

Early intervention in individuals with Autism Spectrum Disorder (ASD) can improve core and associated symptoms and facilitate skills that increase social opportunities. However, determining effective intervention success in this population, and the mechanisms that produce it, is currently restricted to observable behavior. The need of therapy assessment metrics beyond traditional behavioral criteria, led to the use of physiological signals for capturing child-therapist internal dynamics during an intervention session. Internal physiological states were measured through Electrodermal Activity (EDA) and modeled in relation to observed self- and co-regulatory behaviors. A common measure of EDA, Skin Conductance Response (SCR), was the primary signal of interest and assumed to form a non-homogeneous Poisson Process whose rate function is determined by observed regulatory behaviors. Through likelihood and residual goodness of fit analysis, statistical tests and classification tasks, our results indicate that SCR changes and observable behavior in child-therapist dyads are temporally associated and the estimated model parameters can be linked to the types of regulation stimuli.

Individuals diagnosed with Autism Spectrum Disorders (ASD) often have challenging behaviors (CB's), such as self-injury or emotional outbursts, which can negatively impact the quality of life of themselves and those around them. Recent advances in mobile and ubiquitous technologies provide an opportunity to efficiently and accurately capture important information preceding and associated with these CB's. The ability to obtain this type of data will help with both intervention and behavioral phenotyping efforts. Through collaboration with behavioral scientists and therapists, we identified relevant design requirements and created an easy-to-use mobile application for collecting, labeling, and sharing in-situ behavior data in individuals diagnosed with ASD. Furthermore, we have released the application to the community as an open-source project so it can be validated and extended by other researchers.

Collection and analysis of ultra-dense, longitudinal observational data of child behavior in natural, ecologically valid, non-laboratory settings holds significant promise for advancing the understanding of child development and developmental disorders such as autism. To this end, we created the Speechome Recorder - a portable version of the embedded audio/video recording technology originally developed for the Human Speechome Project - to facilitate swift, cost-effective deployment in home environments. Recording child behavior daily in these settings will enable detailed study of developmental trajectories in children from infancy through early childhood, as well as typical and atypical dynamics of communication and social interaction as they evolve over time. Its portability makes possible potentially large-scale comparative study of developmental milestones in both neurotypical and developmentally delayed children. In brief, the Speechome Recorder was designed to reduce cost, complexity, invasiveness and privacy issues associated with naturalistic, longitudinal recordings of child development.

Widespread use of affective sensing in healthcare applications has been limited due to several practical factors, such as lack of comfortable wearable sensors, lack of wireless standards, and lack of low-power affordable hardware. In this paper, we present a new low-cost, low-power wireless sensor platform implemented using the IEEE 802.15.4 wireless standard, and describe the design of compact wearable sensors for long-term measurement of electrodermal activity, temperature, motor activity, and photoplethysmography. We also illustrate the use of this new technology for continuous long-term monitoring of autonomic nervous system and motion data from active infants, children, and adults. We describe several new applications enabled by this system, discuss two specific wearable designs for the wrist and foot, and present sample data.

Individuals with Autism Spectrum Disorders (ASD) frequently engage in stereotyped and repetitive motor movements. Automatically detecting these movements in real-time using comfortable, miniature wireless sensors could advance autistic research and enable new intervention tools for the classroom that help children and their caregivers monitor and cope with this potentially problematic class of behavior. We present activity recognition results for stereotypical hand flapping and body rocking using data collected from six children with ASD repeatedly observed in both laboratory and classroom settings. In the classroom, an overall recognition accuracy of 88.6% (TP: 0.85; FP: 0.08) was achieved using three sensors. Challenges encountered when applying machine learning to this domain, as well as implications for the development of real-time classroom interventions and research tools, are discussed.

Individuals on the autism spectrum often have difficulties producing intelligible speech with either high or low speech rate, and atypical pitch and/or amplitude affect.In this study, we present a novel intervention towards customizing speech enabled games to help them produce intelligible speech.In this approach, we clinically and computationally identify the areas of speech production difficulties of our participants.We provide an interactive and customized interface for the participants to meaningfully manipulate the prosodic aspects of their speech.Over the course of 12 months, we have conducted several pilots to set up the experimental design, developed a suite of games and audio processing algorithms for prosodic analysis of speech.Preliminary results demonstrate our intervention being engaging and effective for our participants.

Stereotypical motor movements are one of the most common and least understood behaviors occurring in individuals with Autism Spectrum Disorder (ASD). Problems with traditional methods for measuring movement stereotypy make it difficult to accurately determine when and why these behaviors occur. The current research overcomes previous measurement problems by utilizing wireless accelerometers and pattern recognition software to automatically and reliably detect stereotypical motor movements such as body rocking and hand flapping in children with ASD.

Many first-hand accounts from individuals on the autism spectrum highlight the challenges inherent in processing high-speed, complex, and unpredictable social information such as nonverbal cues in real-time. In this paper, we describe iSET (Interactive Social- Emotional Toolkit): a suite of wearable technologies that can enable individuals on the autism spectrum to capture, quantify, analyze, and share aspects of their own and others' social interactions in order to facilitate learning and generalization of social-emotional communication. We present several applications of iSET we are currently piloting, and highlight the technical, usability, and experimental challenges we face in designing, implementing, and evaluating this technology.

Many first-hand accounts from individuals diagnosed with autism spectrum disorders (ASD) highlight the challenges inherent in processing high-speed, complex, and unpredictable social information such as facial expressions in real-time. In this paper, we describe a new technology aimed at helping people capture, analyze, and reflect on a set of social-emotional signals communicated by facial and head movements in live social interaction that occurs with their everyday social companions. We describe our development of a new combination of hardware using a miniature camera connected to an ultramobile PC together with custom software developed to track, capture, interpret, and intuitively present various interpretations of the facial-head movements (e.g., presenting that there is a high probability the person looks "confused"). This paper describes this new technology together with the results of a series of pilot studies conducted with adolescents diagnosed with ASD who used the technology in their peer-group setting and contributed to its development via their feedback.