Monitoring daily activities is essential for home service robots to take care of the older adults who live alone in their homes. In this article, we proposed a sound-based human activity monitoring (SoHAM) framework by recognizing sound events in a home environment. First, the method of context-aware sound event recognition (CoSER) is developed, which uses contextual information to disambiguate sound events. The locational context of sound events is estimated by fusing the data from the distributed passive infrared (PIR) sensors deployed in the home. A two-level dynamic Bayesian network (DBN) is used to model the intratemporal and intertemporal constraints between the context and the sound events. Second, dynamic sliding time window-based human action recognition (DTW-HaR) is developed to estimate active sound event segments with their labels and durations, then infer actions and their durations. Finally, a conditional random field (CRF) model is proposed to predict human activities based on the recognized action, location, and time. We conducted experiments in our robot-integrated smart home (RiSH) testbed to evaluate the proposed framework. The obtained results show the effectiveness and accuracy of CoSER, action recognition, and human activity monitoring.Note to Practitioners—This article is motivated by the goal to develop companion robots that can assist older adults living alone. Among many capabilities, monitoring human daily activities is an essential one for such robots. Though computer vision or wearable sensors-based methods have been developed by other researchers, they are not practical due to the privacy concern and intrusiveness. Sound-based daily activity recognition can address these concerns and offer a viable solution. In this regard, our proposed method adopts microphones on the robot and a small set of motion sensors distributed in the home. The proposed theoretical framework was tested in a small-scale mock-up apartment with promising results. Before such companion robots can be deployed to real homes for elderly care, there is a need to improve the robustness of the algorithms. More thorough tests in various realistic home environments should be conducted to fully evaluate the performance of the robots. In addition, privacy concern related to audio capture should be further mitigated.

In this paper, we propose a novel online algorithm for motion similarity measurements during human-robot interaction (HRI). Specifically, we formulate a Segment-based Online Dynamic Time Warping (SODTW) algorithm that can be used for understanding of repeated and cyclic human motions, in the context of rehabilitation or social interaction. The algorithm can estimate both the human-robot motion similarity and the time delay to initiate motion and combine these values as a metric to adaptively select appropriate robot imitation repertoires. We validated the algorithm offline by post-processing experimental data collected from a cohort of 55 subjects during imitation episodes with our social robot Zeno. Furthermore, we implemented the algorithm online on Zeno and collected further experimental results with 13 human subjects. These results show that the algorithm can reveal important features of human movement including the quality of motion and human reaction time to robot stimuli. Moreover, the robot can adapt to appropriate human motion speeds based on similarity measurements calculated using this algorithm, enabling future adaptive rehabilitation interventions for conditions such as Autism Spectrum Disorders (ASD).

Accurate registration of cardiac first-pass magnetic resonance imaging (FP-MRI) is fundamental for precise analysis of myocardial perfusion. In this paper, we introduce and validate a new framework for accurate registration of the segmented left ventricle (LV) wall on cardiac FP-MRI. Due to the continuous physiological motion of the heart that causes the LV wall to change shape significantly and to move within and through the image plane, we developed a new methodology for 2D FP-MRI nonrigid registration that includes: (i) global target-to-reference frame-to-frame alignment based on the maximization of the normalized mutual information (NMI); (ii) local alignment based on using a B-splines transformation model that maximizes a similarity function that accounts for 1st- and 2nd-order NMI between the globally aligned frames followed by (iii) a refinement step that is based on deforming each pixel of the target wall over evolving closed equi-spaced contours (iso-contours) to closely match the reference wall. Respective iso-contours in both reference and target frames are matched based on solving the Laplace equation. We have tested our framework on both synthetic phantoms and 20 in-vivo data sets that have been collected from patients with ischemic damage from heart attacks, who are undergoing a novel myoregeneration therapy.

This paper presents an overview of our work on the investigation of an affect-sensitive system to be applied in future autism intervention. A physiology-based affect-inference and adaptation framework was proposed, which could endow the assistive intervention technology with the capability of detecting the affective states of a child with autism spectrum disorder (ASD) and responding to them accordingly. Given the importance of affective cues in human-machine interaction and its significant role in autism intervention practice, this work marks an important step towards intelligent intervention systems that embody human-like functionality - affect recognition and adaptation. To account for the spectrum nature of autism and the differences of emotional expression, an individual-specific approach was employed for affective modeling. Two computer-based cognitive tasks were designed for eliciting target affective states considered important in autism intervention. Experimental results on real-time affect recognition and adaptation are presented based on a study of 6 children with ASD in a proof-of-concept experiment (i.e., robot-based basketball game). The preliminary results demonstrated that such an affect-sensitive adaptive system could hold promise for computer/robot-assisted autism intervention.

Investigation into robot-assisted intervention for children with autism spectrum disorder (ASD) has gained momentum in recent years. Therapists involved in interventions must overcome the communication impairments generally exhibited by children with ASD by adeptly inferring the affective cues of the children to adjust the intervention accordingly. Similarly, a robot must also be able to understand the affective needs of these children-an ability that the current robot-assisted ASD intervention systems lack-to achieve effective interaction that addresses the role of affective states in human-robot interaction and intervention practice. In this paper, we present a physiology-based affect-inference mechanism for robot-assisted intervention where the robot can detect the affective states of a child with ASD as discerned by a therapist and adapt its behaviors accordingly. This paper is the first step toward developing ldquounderstandingrdquo robots for use in future ASD intervention. Experimental results with six children with ASD from a proof-of-concept experiment (i.e., a robot-based basketball game) are presented. The robot learned the individual liking level of each child with regard to the game configuration and selected appropriate behaviors to present the task at his/her preferred liking level. Results show that the robot automatically predicted individual liking level in real time with 81.1% accuracy. This is the first time, to our knowledge, that the affective states of children with ASD have been detected via a physiology-based affect recognition technique in real time. This is also the first time that the impact of affect-sensitive closed-loop interaction between a robot and a child with ASD has been demonstrated experimentally.

This paper presents a novel affect-sensitive human-robot interaction framework for rehabilitation of children with autism spectrum disorder (ASD). The overall aim is to enable the robot to detect and respond to the affective cues of the children in order to help them explore social interaction dynamics in a gradual and adaptive manner. The first part of the proposed framework, namely the 'affect recognition' module is developed in detail in this paper. Two tasks are designed to elicit the affective states of liking, anxiety, and engagement that are considered important in autism rehabilitation. Affective cues are inferred from psychophysiological analysis that uses subjective reports of the affective states from a therapist, a parent, and the child himself/herself. Comprehensive physiological indices are investigated that may correlate with the affective states of children with ASD. A support vector machines based affect recognizer is designed that yielded reliable prediction with approximately 83% success when using the therapist's reports. This is the first time, to our knowledge, such a human-robot interaction framework for autism rehabilitation is proposed. This is also the first time that the affective states of children with ASD have been experimentally detected via physiology-based affect recognition technique.

This paper presents a novel affect-sensitive human-robot interaction framework for rehabilitation of children with autism spectrum disorder (ASD) where the robot can detect the affective cues of the children implicitly and response to them appropriately. Psychophysiological analysis is performed that uses subjective reports of the affective states from a clinical observer. Comprehensive physiological indices are investigated that may correlate with the affective states of children with ASD. A robot uses a support vector machines based affect model to detect the affective cues. A reinforcement learning based adaptation mechanism is employed to allow the robot to adjust its behaviors autonomously as a function of the predicted children's affective state. Four adolescents diagnosed with high-functioning autism participated in the experiments. This is the first time, to our knowledge, that the affective states of children with ASD have been detected via physiology-based affective modeling technique in real-time. This is also the first time that impact of affect-sensitive interaction between a robot and children with ASD in closed loop has been demonstrated experimentally.