Many older adults prefer to stay in their own homes and age-in-place. However, physical and cognitive limitations in independently completing activities of daily living (ADLs) requires older adults to receive assistive support, often necessitating transitioning to care centers. In this paper, we present the development of a novel deep learning human activity recognition and classification architecture capable of autonomously identifying ADLs in home environments to enable long-term deployment of socially assistive robots to aid older adults. Our deep learning architecture is the first to use multimodal inputs to create an embedding vector approach for classifying and monitoring multiple ADLs. It uses spatial mid-fusion to combine geometric, motion and semantic features of users, environments, and objects to classify and track ADLs. We leverage transfer learning to extract generic features using the early layers of deep networks trained on large datasets to apply our architecture to various ADLs. The embedding vector enables identification of unseen ADLs and determines intra-class variance for monitoring user ADL performance. Our proposed unique architecture can be used by socially assistive robots to promote reablement in the home via autonomously supporting the assistance of varying ADLs. Extensive experiments show improved classification accuracy compared to unimodal/dual-modal models and the ADL embedding space also incorporates the ability to distinctly identify and track seen and unseen ADLs.

Individuals living with cognitive impairments are faced with unique challenges in completing important activities of daily living such as dressing. In this paper, we present the first socially assistive robot-wearable sensors system to provide dressing assistance through social human-robot interactions. A novel robot-wearable architecture is development to classify, prompt and provide feedback on user dressing actions. Namely, strain sensor based smart clothing on the user are used for joint angle mapping, which are then classified into different dressing steps. The robot uses a MAXQ hierarchical learning method to learn assistive behaviors to aid a user with the sequence of dressing steps. Experiments were validated the performance of the joint angle mapping model, dressing action classifier, and behavior adaptation modules as well as the overall system for dressing assistance.