Group-based dance therapy can provide physical health, social and cognitive benefits for older adults, as well as promote mental health. In this article, we present a novel Human–Robot Interaction (HRI) study that explores how robot-facilitated group-based dance therapy sessions influence the affect, engagement, and enjoyment of a group of older adults interacting with a socially assistive robot. Furthermore, we investigate how robot embodiment size impacts group experiences during HRI. A human-size robot and a toy-size robot both facilitated group-based therapy sessions with older adults at a long-term care home over a 2-month period. Extensive results showed that robot dance therapy can effectively increase group valence and engagement through repeated interaction sessions, while maintaining high levels of group enjoyment. There were no statistically significant differences found in group affect, engagement, and enjoyment among these two different robot embodiment sizes.

Mobile robots in unknown cluttered environments with irregularly shaped obstacles often face energy and communication challenges which directly affect their ability to explore these environments. Existing heuristic and learning-based map prediction methods are unable to generalize to irregular obstacles and uneven terrain, as they rely on single-pass architectures that cannot iteratively refine map predictions or incorporate uncertainty under limited communication and energy constraints. On the other hand, diffusion models perform multi-pass denoising to reconstruct high-fidelity maps from partial observations, enabling accurate predictions in resource constrained settings. In this paper, we introduce a novel deep learning architecture, Confidence-Aware Contrastive Conditional Consistency Model (4CNet), for robot map prediction during decentralized, resource limited multi-robot exploration. 4CNet uniquely incorporates: 1) a conditional consistency model for map prediction in unstructured unknown regions, 2) a contrastive map-trajectory pretraining framework for a trajectory encoder that extracts spatial information from the trajectories of nearby robots during map prediction, and 3) a confidence network to measure the uncertainty of map prediction for effective exploration under resource constraints. We incorporate 4CNet within our proposed robot exploration with map prediction architecture, 4CNet E. We then conduct extensive comparison studies with 4CNet-E and state-of-the-art heuristic and learning methods to investigate both map prediction and exploration performance in environments consisting of irregularly shaped obstacles and uneven terrain. Results showed that 4CNet-E obtained statistically significant higher prediction accuracy and area coverage with varying environment sizes, number of robots, energy budgets, and communication limitations when compared to database and learning-based methods. Hardware experiments were performed and validated the applicability and generalizability of 4CNet-E in both unstructured indoor and real natural outdoor environments.

Service robots in human-centered environments such as hospitals, office buildings, and long-term care homes need to navigate while adhering to social norms to ensure the safety and comfortability of the people they are sharing the space with. Furthermore, they need to adapt to new social scenarios that can arise during robot navigation. In this paper, we present a novel Online Lifelong Vision Language architecture, OLiVia-Nav, which uniquely integrates vision-language models (VLMs) with an online lifelong learning framework for robot social navigation. We introduce a unique distillation approach, Social Context Contrastive Language Image Pre-training (SC-CLIP), to transfer the social reasoning capabilities of large VLMs to a lightweight VLM, in order for OLiVia-Nav to directly encode social and environment context during robot navigation. These encoded embeddings are used to generate and select robot social compliant trajectories. The lifelong learning capabilities of SC-CLIP enable OLiVia-Nav to update the robot trajectory planning overtime as new social scenarios are encountered. We conducted extensive real-world experiments in diverse social navigation scenarios. The results showed that OLiVia-Nav outperformed existing state-of-the-art DRL and VLM methods in terms of mean squared error, Hausdorff loss, and personal space violation duration. Ablation studies also verified the design choices for Ol.Jvia-Nav.

Emotional dance is an engaging and stimulating multimodal social activity involving the display of both body language and music. An interesting area of research is in the investigation of how people perceive the emotions of robots. In particular, how the interaction between several modalities influences emotion perception in human-robot interactions (HRI). In this paper, we present the first study that investigates how robot body movements and music influence human emotion perception with respect to robot emotional dance. Through an online survey, 115 participants rated the emotion expressed by the dancing robot with varying body movements and music in two conditions: 1) robot dancing with music (visual + auditory) and 2) robot dancing without music (only visual). Our results showed that perceived valence is primarily influenced by robot body language and movements, especially with respect to positive valence, regardless of the presence of the music mode. Women also had higher perceived negative valence when observing the negative valence body movements displayed by the robot than men did. Furthermore, music had limited influence on the perception of: 1) valence when negative valence body language is displayed while the music had positive valence, and 2) arousal when the music had negative arousal. Our study provides insights on how to effectively design social HRI when considering human emotional perception of robots.

Many older adults prefer to stay in their own homes and age-in-place. However, physical and cognitive limitations in independently completing activities of daily living (ADLs) requires older adults to receive assistive support, often necessitating transitioning to care centers. In this paper, we present the development of a novel deep learning human activity recognition and classification architecture capable of autonomously identifying ADLs in home environments to enable long-term deployment of socially assistive robots to aid older adults. Our deep learning architecture is the first to use multimodal inputs to create an embedding vector approach for classifying and monitoring multiple ADLs. It uses spatial mid-fusion to combine geometric, motion and semantic features of users, environments, and objects to classify and track ADLs. We leverage transfer learning to extract generic features using the early layers of deep networks trained on large datasets to apply our architecture to various ADLs. The embedding vector enables identification of unseen ADLs and determines intra-class variance for monitoring user ADL performance. Our proposed unique architecture can be used by socially assistive robots to promote reablement in the home via autonomously supporting the assistance of varying ADLs. Extensive experiments show improved classification accuracy compared to unimodal/dual-modal models and the ADL embedding space also incorporates the ability to distinctly identify and track seen and unseen ADLs.

Multirobot coordination for finding multiple users in an environment can be used in numerous robotic applications, including search and rescue, surveillance/monitoring, and activities of daily living assistance. Existing approaches have limited coordination between robots when generating team plans or do not consider user location probability within these plans. This results in long searches and robots potentially revisiting the same locations in succession. In this article, we present a novel multirobot person search system to generate search plans for multirobot teams to find multiple dynamic users before a deadline. Our approach is unique in that it simultaneously considers the search actions of all robots and user location probabilities when generating team plans, where user location probabilities are represented as conditional spatial-temporal probability density functions. We model this multirobot person search problem as a two-stage optimization problem to maximize the expected number of users found before the deadline. Stage 1 solves the action selection problem to determine a set of team actions, and the second stage solves the action allocation problem to distribute these actions amongst the robots. Namely, in stage 1, a novel conditional multiperiod multiknapsack problem is modeled as a min-flow graph solved sequentially by the Bellman-Ford shortest path algorithm. Stage 2 is a variant of the min-max multitraveling salesperson problem which models the environment topology as a search region network and search times selected by the previous stage. This stage is solved by a novel fuzzy clustering method. Numerous experiments comparing our proposed method to other existing approaches with varying environment sizes, search durations, and the number of users showed that our approach was able to find more target users before a defined deadline.

The rapid spread of COVID-19 around the globe has increased the need to adopt autonomous social robots within our healthcare systems. In particular, socially assistive robots can help to improve the day-to-day functioning of our healthcare facilities including long-term care, while keeping residents and staff safe by performing repetitive tasks such as health screening. In this paper, we present the first human-robot interaction study with an autonomous multi-task socially assistive robot used for non-contact screening in long-term care homes. The robot monitors temperature, checks for face masks, and asks screening questions to minimize human-to-human contact. We investigated staff perceptions of 7 attributes: screening experience without and with the robot, efficiency, cognitive attitude, freeing up staff, safety, affective attitude, and intent to use the robot. Furthermore, we investigated the influence of demographics on these attributes. Study results show that, overall, staff rated these attributes high for the screening robot, with a statistically significant increase in cognitive attitude and safety after interacting with the robot. Differences between gender and occupation were also determined. Our study highlights the potential application of an autonomous screening robot for long-term care homes.

Individuals living with cognitive impairments are faced with unique challenges in completing important activities of daily living such as dressing. In this paper, we present the first socially assistive robot-wearable sensors system to provide dressing assistance through social human-robot interactions. A novel robot-wearable architecture is development to classify, prompt and provide feedback on user dressing actions. Namely, strain sensor based smart clothing on the user are used for joint angle mapping, which are then classified into different dressing steps. The robot uses a MAXQ hierarchical learning method to learn assistive behaviors to aid a user with the sequence of dressing steps. Experiments were validated the performance of the joint angle mapping model, dressing action classifier, and behavior adaptation modules as well as the overall system for dressing assistance.

Persuasion is a fundamental aspect of how people interact with each other. As robots become integrated into our daily lives and take on increasingly social roles, their ability to persuade will be critical to their success during human-robot interaction (HRI). In this article, we present a novel HRI study that investigates how a robot's persuasive behavior influences people's decision making. The study consisted of two small social robots trying to influence a person's answer during a jelly bean guessing game. One robot used either an emotional or logical persuasive strategy during the game, while the other robot displayed a neutral control behavior. The results showed that the Emotion strategy had significantly higher persuasive influence compared to both the Logic and Control conditions. With respect to participant demographics, no significant differences in influence were observed between age or gender groups; however, significant differences were observed when considering participant occupation/field of study (FOS). Namely, participants in business, engineering, and physical sciences fields were more influenced by the robots and aligned their answers closer to the robot's suggestion than did those in the life sciences and humanities professions. The discussions provide insight into the potential use of robot persuasion in social HRI task scenarios; in particular, considering the influence that a robot displaying emotional behaviors has when persuading people.

Swarm robots can achieve effective task execution via closed-loop motion control. However, such a goal can only be realized through accurate localization of the swarm. Past approaches have focused on addressing this issue using external sensors, static sensor networks, or through active localization—requirements that may restrict the motion of the swarm or may not be achievable in practice. We present a tether-based strategy that achieves closed-loop swarm-motion control by using a secondary team of mobile sensors. These sensors form a wireless tether that allows the swarm to indirectly sense a home base or a landmark, and to compensate for the accumulated motion errors via a closed-loop control strategy. The proposed strategy is the first to use a tether of mobile sensors that can dynamically reshape and reconnect to various points in the environment to achieve closed-loop motion control. The novelty of the strategy is in its ability to adapt to any swarm motion considered, and to be applied to swarms with limited sensing capabilities and knowledge of their environment. The performance of the proposed strategy was validated through extensive experiments.

The COVID-19 pandemic is one of the most challenging healthcare crises during the 21stcentury. As the virus continues to spread on a global scale, the majority of efforts have been on the development of vaccines and the mass immunization of the public. While the daily case numbers were following a decreasing trend, the emergent of new virus mutations and variants still pose a significant threat. As economies start recovering and societies start opening up with people going back into office buildings, schools, and malls, we still need to have the ability to detect and minimize the spread of COVID-19. Individuals with COVID-19 may show multiple symptoms such as cough, fever, and shortness of breath. Many of the existing detection techniques focus on symptoms having the same equal importance. However, it has been shown that some symptoms are more prevalent than others. In this paper, we present a multimodal method to predict COVID-19 by incorporating existing deep learning classifiers using convolutional neural networks and our novel probability-based weighting function that considers the prevalence of each symptom. The experiments were performed on an existing dataset with respect to the three considered modes of coughs, fever, and shortness of breath. The results show considerable improvements in detection of COVID-19 using our weighting function when compared to an equal weighting function.

For social robots to effectively engage in human-robot interaction (HRI), they need to be able to interpret human affective cues and to respond appropriately via display of their own emotional behavior. In this article, we present a novel multimodal emotional HRI architecture to promote natural and engaging bidirectional emotional communications between a social robot and a human user. User affect is detected using a unique combination of body language and vocal intonation, and multimodal classification is performed using a Bayesian Network. The Emotionally Expressive Robot utilizes the user's affect to determine its own emotional behavior via an innovative two-layer emotional model consisting of deliberative (hidden Markov model) and reactive (rule-based) layers. The proposed architecture has been implemented via a small humanoid robot to perform diet and fitness counseling during HRI. In order to evaluate the Emotionally Expressive Robot's effectiveness, a Neutral Robot that can detect user affects but lacks an emotional display, was also developed. A between-subjects HRI experiment was conducted with both types of robots. Extensive results have shown that both robots can effectively detect user affect during the real-time HRI. However, the Emotionally Expressive Robot can appropriately determine its own emotional response based on the situation at hand and, therefore, induce more user positive valence and less negative arousal than the Neutral Robot.

Patient-specific control and training on lower body exoskeletons can help improve a user's gait during post-stroke rehabilitation by increasing their amount of participation and motor learning. Traditionally, adaptive control techniques have been used to provide personalization and synchronization with exoskeleton users, but they require predefined dynamics models of the user and exoskeleton. However, these models can be difficult to accurately define due to the complexity of the human-robot interaction. Most recently deep reinforcement learning techniques have shown potential to effectively learn control schemes without the need for system dynamics models. In this paper, we present for the first time an end-to-end model-free deep reinforcement learning method for an exoskeleton that can learn to follow a desired gait pattern, while considering a user's existing gait pattern and being robust to their perturbations and interactions. We demonstrate the effectiveness of our proposed method for user personalization of gait training in simulated experiments.

Robot search for multiple dynamic users within a multi-room environment is important for social robots to find and engage in various human-robot interaction scenarios with these users. In this paper, we present a novel autonomous person search technique for a robot finding a group of dynamic users before a deadline. The uniqueness of our approach is that unlike existing robot search methods, we consider activity information to predict where, when, and for how long a user will be in a specific room. This allows for the generation of search plans without any assumption on the frequency of user movements. We represent our search problem as an extension of the orienteering problem (OP), which we define herein as the robot person search OP (PSOP). User activity information is represented as spatial-temporal user activity probability density functions (APDFs). We solve the PSOP using APDFs to generate a search plan to maximize the expected number of users found before the deadline. The solution of the PSOP is obtained in two steps. First, by solving a variant of the multiperiod knapsack problem to determine which rooms should be searched and for how long these rooms should be searched. Then, we solve the traveling salesman problem to obtain the order in which to search these rooms. Experiments were conducted to validate the performance of our robot search method in finding different numbers of multiple dynamic users for varying environment sizes and search durations. We also compared our method with two coverage planners and a Markov decision process planner. On average, our planner found more users than the other planners for a variety of scenarios. Finally, we performed experiments that introduced uncertainty into both the APDFs as well as during the search to validate the robustness of our overall approach.

Locating a mobile target, untrackable in real-time, is pertinent to numerous time-critical applications, such as wilderness search and rescue. This paper proposes a hybrid approach to this dynamic problem, where both static and mobile sensors are utilized for the goal of detecting a target. The approach is novel in that a team of robots utilized to deploy a static-sensor network also actively searches for the target via on-board sensors. Synergy is achieved through: 1) optimal deployment planning of static-sensor networks and 2) optimal routing and motion planning of the robots for the deployment of the network and target search. The static-sensor network is planned first to maximize the likelihood of target detection while ensuring (temporal and spatial) unbiasedness in target motion. Robot motions are, subsequently, planned in two stages: 1) route planning and 2) trajectory planning. In the first stage, given a static-sensor network configuration, robot routes are planned to maximize the amount of spare time available to the mobile agents/sensors, for target search in between (just-in-time) static-sensor deployments. In the second stage, given robot routes (i.e., optimal sequences of sensor delivery locations and times), the corresponding robot trajectories are planned to make effective use of any spare time the mobile agents may have to search for the target. The proposed search strategy was validated through extensive simulations, some of which are given in detail here. An analysis of the method's performance in terms of target-search success is also included.

Dressing is an Activity of Daily Living (ADL) that can be difficult to do for individuals living with cognitive disorders and can, therefore, negatively impact their quality of life. Our research focuses on the development of an assistive robot and smart clothing system to aid a user with this ADL. In this paper, we present our autonomous Clothing Perception System that uniquely incorporates smart sensors into clothing in order to perceive if a person has worn the clothes correctly. Four different dressing states can be identified: correctly worn; partially worn; backwards; or inverted (inside out). Our novel system uses a combination of capacitive sensors, contact switches, an infrared LED and an RGB-D sensor to determine the dressing state. The multi-modal sensing system was integrated into a collared shirt and tested to verify its performance.Results with different individuals putting on the shirt showed that the system was able to perceive the four distinct dressing states for all of them.

Regular exercise has immediate and long-term benefits for people of all ages. Maintaining an adequate amount of daily exercise is important to overall health and wellbeing. Our research focuses on the development of a socially assistive robot, Salt, to facilitate different upper body exercises. During the exercises, the robot is uniquely able to autonomously detect a user's affect and engagement as well as measure their heart rate to prevent overexertion. A robot emotion model using an n th order Markov Chain is used to determine the robot's appropriate emotions during interactions based on user affect and engagement, and its own emotion history. Human-robot interaction experiments were conducted to investigate perceived usefulness and acceptance. The results showed that most users were engaged and had positive valence towards the robot during the interactions. Post-experiment questionnaire results also showed they were able to detect the robot's emotions and enjoyed interacting with it.

Time-phased sensor-network deployment refers to the delivery of a set of sensors to their predetermined locations at exact times by a fleet of vehicles. Applications for such network deployments include wilderness search and rescue (WiSAR) and wildfire monitoring, where desirable resource management would imply allowing the vehicles to perform other tasks between deliveries. The goal of this paper is, thus, to formulate and solve a vehicle-routing problem (VRP) for such just-in-time time-phased sensor-network deployments. The proposed optimization method for the modified VRP outlined herein has two primary novelties: 1) the consideration of spare time as the objective function and 2) the use of a targeted local-search (LS) method. The spare-time objective function was formulated to address the uniqueness of the modified routing problem at hand. The targeted LS algorithm, on the other hand, was developed to tangibly improve the efficiency of the search for the optimal values of the chosen objective function. The proposed vehicle-route-planning method was validated via a range of simulated WiSAR scenarios, some of which are included herein. The robustness of the method to variations in problem parameters was also investigated.

Socially assistive robots can autonomously provide activity assistance to vulnerable populations, including those living with cognitive impairments. To provide effective assistance, these robots should be capable of displaying appropriate behaviors and personalizing them to a user's cognitive abilities. Our research focuses on the development of a novel robot learning architecture that uniquely combines learning from demonstration ( LfD ) and reinforcement learning ( RL ) algorithms to effectively teach socially assistive robots personalized behaviors. Caregivers can demonstrate a series of assistive behaviors for an activity to the robot, which it uses to learn general behaviors via LfD . This information is used to obtain initial assistive state-behavior pairings using a decision tree. Then, the robot uses an RL algorithm to obtain a policy for selecting the appropriate behavior personalized to the user's cognition level. Experiments were conducted with the socially assistive robot Casper to investigate the effectiveness of our proposed learning architecture. Results showed that Casper was able to learn personalized behaviors for the new assistive activity of tea-making, and that combining LfD and RL algorithms significantly reduces the time required for a robot to learn a new activity.

We investigate Constraint Programming and Planning Domain Definition Language-based technologies for planning and scheduling multiple robots in a retirement home environment to assist elderly residents. Our robotics problem and investigation into proposed solution approaches provide a real world application of planning and scheduling, while highlighting the different modeling assumptions required to solve such a problem. This information is valuable to the planning and scheduling community as it provides insight into potential application avenues, in particular for robotics problems. Based on empirical results, we conclude that a constraint-based scheduling approach, specifically a decomposition using constraint programming, provides the most promising results for our application.

This paper discusses the time-phased deployment of wireless sensor networks, applied to surveillance areas growing in time. The focus herein is on the planning of the time-efficient delivery of static sensors to their designated nodes, given a network configuration. The novelty of the proposed strategy is in that it determines optimal delivery plans for spatio-temporally constrained static-sensor networks using multi-robot teams. The proposed sensor delivery planning strategy starts with an already determined (optimal) network plan specified by sensor placement locations (i.e., nodes) and deployment times. Thus, the goal at hand is to determine the optimal routes for the robots delivering the sensors to their intended locations just-in-time. The travel routes are, thus, determined to maximize spare time for the robots between the nodes. The problem is similar to the multiple travelling salesperson problem, but, with temporal constraints. Namely, sensors must be delivered to their designated nodes at designated (optimized) times in order to maintain the optimal deployment of the network configuration. Furthermore, the strategy is designed to be adaptive to new information that can become available during the search for the mobile target, allowing for replanning of the sensor network (i.e., new sensors locations and new deployment times). Numerous simulated experiments were conducted to validate the proposed strategy.

This paper investigates three different technologies for solving a planning and scheduling problem of deploying multiple robots in a retirement home environment to assist elderly residents. The models proposed make use of standard techniques and solvers developed in AI planning and scheduling, with two primary motivations. First, to find a planning and scheduling solution that we can deploy in our real-world application. Second, to evaluate planning and scheduling technology in terms of the ``model-and-solve'' functionality that forms a major research goal in both domain-independent planning and constraint programming. Seven variations of our application are studied using the following three technologies: PDDL-based planning, time-line planning and scheduling, and constraint-based scheduling. The variations address specific aspects of the problem that we believe can impact the performance of the technologies while also representing reasonable abstractions of the real world application. We evaluate the capabilities of each technology and conclude that a constraint-based scheduling approach, specifically a decomposition using constraint programming, provides the most promising results for our application. PDDL-based planning is able to find mostly low quality solutions while the timeline approach was unable to model the full problem without alterations to the solver code, thus moving away from the model-and-solve paradigm. It would be misleading to conclude that constraint programming is ``better'' than PDDL-based planning in a general sense, both because we have examined a single application and because the approaches make different assumptions about the knowledge one is allowed to embed in a model. Nonetheless, we believe our investigation is valuable for AI planning and scheduling researchers as it highlights these different modelling assumptions and provides insight into avenues for the application of AI planning and scheduling for similar robotics problems. In particular, as constraint programming has not been widely applied to robot planning and scheduling in the literature, our results suggest significant untapped potential in doing so.

For social robots to be successfully integrated and accepted within society, they need to be able to interpret human social cues that are displayed through natural modes of communication. In particular, a key challenge in the design of social robots is developing the robot's ability to recognize a person's affective states (emotions, moods, and attitudes) in order to respond appropriately during social human-robot interactions (HRIs). In this paper, we present and discuss social HRI experiments we have conducted to investigate the development of an accessibility-aware social robot able to autonomously determine a person's degree of accessibility (rapport, openness) toward the robot based on the person's natural static body language. In particular, we present two one-on-one HRI experiments to: 1) determine the performance of our automated system in being able to recognize and classify a person's accessibility levels and 2) investigate how people interact with an accessibility-aware robot which determines its own behaviors based on a person's speech and accessibility levels.

Millirobots have increasingly become popular over the past several years, especially for swarm-behavior studies, allowing researchers to run experiments with a large number of units in limited workspaces. However, as these robots have become smaller in size, their sensory capabilities and battery life have been reduced. A number of these have also been customized, with few off-the shelf components, exhibiting integral (i.e., non-modular) designs. In response to the above concerns, this paper presents a novel open-source millirobot with a modular design based on the use of easily sourced elements and off-the-shelf components. The proposed milli-robot-Toronto (mROBerTO), is a 16×16 mm2robot with a variety of sensors (including proximity, IMU, compass, ambient light, and camera). mROBerTO is capable of formation control using an IR emitter and detector add-on. It can also communicate via Bluetooth Smart, ANT+, or both concurrently. It is equipped with an ARM processor for handling complex tasks and has a flash memory of 256 KB with over-the-air programming capability.

Group-based recreational activities have shown to have a number of health benefits for people of all ages. The handful of social robots designed to facilitate such activities are currently only able to implement a priori known recreational activities that have been pre-programmed by human experts. Once deployed in their intended facility, these robots are not able to learn new activities from non-expert humans. In this paper, we present the development of a novel learning from demonstration (LfD) system architecture for a social robot in order for it to learn from non-expert teachers the structure of an activity and monitor the execution of the new activity. In order to obtain user compliance, personalized persuasive strategies are also learned by the robot to use while implementing the activity during human-robot interactions (HRI) with the intended users. The architecture has been integrated into our socially assistive robot Tangy to learn the group-based activity Bingo. System performance experiments were conducted with Tangy to first learn to facilitate Bingo from non-expert teachers and then use the learned activity to physically facilitate Bingo with multiple users. The results showed Tangy was able to effectively and efficiently learn the new Bingo activity structure as well as personalize its persuasive strategies to individual users in order to obtain activity compliance.

The objective of a socially assistive robot is to create a close and effective interaction with a human user for the purpose of giving assistance. In particular, the social interaction, guidance, and support that a socially assistive robot can provide a person can be very beneficial to patient-centered care. However, there are a number of research issues that need to be addressed in order to design such robots. This paper focuses on developing effective emotion-based assistive behavior for a socially assistive robot intended for natural human-robot interaction (HRI) scenarios with explicit social and assistive task functionalities. In particular, in this paper, a unique emotional behavior module is presented and implemented in a learning-based control architecture for assistive HRI. The module is utilized to determine the appropriate emotions of the robot to display, as motivated by the well-being of the person, during assistive task-driven interactions in order to elicit suitable actions from users to accomplish a given person-centered assistive task. A novel online updating technique is used in order to allow the emotional model to adapt to new people and scenarios. Experiments presented show the effectiveness of utilizing robotic emotional assistive behavior during HRI scenarios.

This paper presents a novel strategy for the on-line planning of optimal motion-paths for a team of autonomous ground robots engaged in wilderness search and rescue (WiSAR). The proposed strategy, which forms part of an overall multirobot coordination (MRC) methodology, addresses the dynamic nature of WiSAR by: 1) planning initial, time-optimal, and piecewise polynomial paths for all robots; 2) implementing and regularly evaluating the optimality of the paths through a set of checks that gauge feasibility of path-completion within the available time; and 3) replanning paths, on-line, whenever deemed necessary. The fundamental principle of maintaining the optimal deployment of the robots throughout the search guides the MRC methodology. The proposed path-planning strategy is illustrated through a simulated realistic WiSAR example, and compared to an alternative, nonprobabilistic approach.

In this paper we address the planning problem of a robot searching for multiple residents in a retirement home in order to remind them of an upcoming multi-person recreational activity before a given deadline. We introduce a novel Multi-User Schedule Based (M-USB) Search approach which generates a high-level-plan to maximize the number of residents that are found within the given time frame. From the schedules of the residents, the layout of the retirement home environment as well as direct observations by the robot, we obtain spatio-temporal likelihood functions for the individual residents. The main contribution of our work is the development of a novel approach to compute a reward to find a search plan for the robot using: 1) the likelihood functions, 2) the availabilities of the residents, and 3) the order in which the residents should be found. Simulations were conducted on a floor of a real retirement home to compare our proposed M-USB Search approach to a Weighted Informed Walk and a Random Walk. Our results show that the proposed M-USB Search finds residents in a shorter amount of time by visiting fewer rooms when compared to the other approaches.

In this paper we present the development of a novel multi-user human-robot interaction (HRI) system architecture to allow the social robot Tangy to autonomously plan, schedule and facilitate multi-user activities while considering the users' schedules. During scheduled activities, the robot is able to interact with a group of users by providing both group-based and individualized assistance based on the current state of the activity and the needs of the individual users engaged in the social interactions. Such planning and scheduling of daily activities of a social robot while reasoning about multiple user schedules has not yet been addressed in the literature. Herein, the HRI multi-user activities we consider are a series of Bingo games. System performance experiments presented in the paper validate the use of the proposed multiuser system architecture in: 1) planning and scheduling daily Bingo games for Tangy to facilitate while considering the individual schedules of the users, and 2) determining the appropriate behaviors of the robot with respect to individuals and groups of people while providing game reminders prior to a Bingo game starting and also while facilitating the game itself.

Our research focuses on the development of a socially assistive robot to provide cognitive and social stimulation during meal-time scenarios in order to promote proper nutrition amongst the elderly. In this paper, we present the design of a novel automated affect recognition and classification system that will allow the robot to interpret natural displays of affective human body language during such one-on-one assistive scenarios. Namely, we identify appropriate body language features and learning-based classifiers that can be utilized for accurate affect estimation. A robot can then utilize this information in order to determine its own appropriate responsive behaviors to keep people engaged in this crucial activity. One-on-one assistive meal-time experiments were conducted with the robot Brian 2.1 and elderly participants at a long-term care facility. The results showed the potential of utilizing the automated affect recognition and classification system to identify and classify natural affective body language features into valence and arousal values using learning-based classifiers. The elderly users displayed a number of affective states, further motivating the use of the affect estimation system.

As older adults age, they are more likely to reside in long-term care facilities due to the decline in cognitive and/or physical abilities that prevent them from living independently. With a rapidly aging population there is an increasing demand on long-term care facilities to care for older adults. Such facilities need to provide medical services, assistance in activities of daily living, and scheduled leisure activities to improve health and quality of life. However, as the need for long-term care is increasing, the care workforce is faced with decreasing numbers of healthcare staff and high turnover rates. Our research focuses on the design of socially assistive robots to plan, schedule, and facilitate social and cognitive interventions for residents in long-term care facilities. In this paper, we investigate the specific design considerations and the impressions of long-term care residents, healthcare professionals, and family members on a socially assistive robot designed to autonomously facilitate cognitively and socially stimulating leisure activities. Thematic analysis of focus group sessions conducted at a long-term care facility with the aforementioned individuals revealed important design considerations for the development and integration of a socially assistive robot in long-term care facilities.

Semi-autonomous control schemes can address the limitations of both teleoperation and fully autonomous robotic control of rescue robots in disaster environments by allowing a human operator to cooperate and share such tasks with a rescue robot as navigation, exploration, and victim identification. In this paper, we present a unique hierarchical reinforcement learning-based semi-autonomous control architecture for rescue robots operating in cluttered and unknown urban search and rescue (USAR) environments. The aim of the controller is to enable a rescue robot to continuously learn from its own experiences in an environment in order to improve its overall performance in exploration of unknown disaster scenes. A direction-based exploration technique is integrated in the controller to expand the search area of the robot via the classification of regions and the rubble piles within these regions. Both simulations and physical experiments in USAR-like environments verify the robustness of the proposed HRL-based semi-autonomous controller to unknown cluttered scenes with different sizes and varying types of configurations.

As people get older, their ability to perform basic self-maintenance activities can be diminished due to the prevalence of cognitive and physical impairments or as a result of social isolation. The objective of our work is to design socially assistive robots capable of providing cognitive assistance, targeted engagement, and motivation to elderly individuals, in order to promote participation in self-maintenance activities of daily living. In this paper, we present the design and implementation of the expressive human-like robot, Brian 2.1, as a social motivator for the important activity of eating meals. An exploratory study was conducted at an elderly care facility with the robot and eight individuals, aged 82--93, to investigate user engagement and compliance during meal-time interactions with the robot along with overall acceptance and attitudes towards the robot. Results of the study show that the individuals were both engaged in the interactions and complied with the robot during two different meal-eating scenarios. A post-study robot acceptance questionnaire also determined that, in general, the participants enjoyed interacting with Brian 2.1 and had positive attitudes towards the robot for the intended activity.

Studies have shown that cognitive and social stimulation is crucial to the overall health of older adults including psychological, cognitive and physical well-being. However, activities to promote such stimulation are often lacking in long-term care facilities. Our work focuses on the use of social robotic technologies to provide person-centered cognitive interventions. Namely, this paper presents an HRI study with the unique human-like socially assistive robot Brian 2.1, in order to investigate the use and acceptability of the expressive human-like robot by older adults living in a longterm care center. Current studies with social robots for the elderly have been mainly directed towards collecting data on the acceptance and use of animal-like robots. Herein, we aim to determine if the robot's human-like assistive and social characteristics result in the elderly having positive attitudes towards the robot as well as accepting it as an interactive cognitive training tool.

In order for robots to effectively engage a person in bi-directional social human-robot interaction (HRI), they need to be able to perceive and respond appropriately to a person's affective state. It has been shown that body language is essential in effectively communicating human affect. In this paper, we present an automated real-time body language recognition and classification system, utilizing the Microsoft®Kinect™ sensor, that determines a person's affect in terms of their accessibility (i.e., openness and rapport) towards a robot during natural one-on-one interactions. Social HRI experiments are presented with our human-like robot Brian 2.0 and a comparison study between our proposed system and one developed with the Kinect™ body pose estimation algorithm verifies the performance of our affect classification system in HRI scenarios.

Recent studies have shown that sustained engagement in cognitively stimulating activities has had positive effects on the cognitive functioning of humans. The objective of our work is to develop an intelligent socially assistive robot that can engage individuals in person-centered cognitively stimulating activities. In this paper, we present the design of a novel learning-based control architecture that enables the robot to act as a social motivator by providing assistance, encouragement and celebration during the course of an activity. A hierarchical reinforcement learning (HRL) approach is used to provide the robot with the ability to: (i) learn appropriate assistive behaviors based on the structure of the activity and (ii) personalize the interaction based on the person's affective state during the activity. Preliminary experiments show that the proposed learning-based control architecture is effective in determining the optimal assistive behaviors of the robot during a memory game interaction.

This paper presents a novel method for the optimal deployment of multi-robot teams for autonomous, coordinated wilderness search and rescue. The new concept of iso-probability curves, used to represent the time-varying prediction of a lost person's probable location within the search area, is utilized to effectively distribute the search effort. The proposed method can be used for initial deployment, as well as subsequent on-line re-deployment to address the dynamic nature of the search for a moving lost person in a growing search area with varying terrain. The modularity of the proposed method allows the user to define and utilize different objective functions and weigh them according to the goal at hand. The two specific objective functions considered in this paper are (minimizing) search time and (maximizing) the probability of success. A simulated realistic wilderness search scenario demonstrates the integration of optimal deployment within the overall search methodology.

Dementia is currently a growing epidemic, bringing forth severe health, social, and economic strains. As an alternative to pharmacological measures, current research supports the effectiveness of using cognitive training interventions to slow the decline of or even improve brain functioning in persons with dementia. However, implementing and sustaining these interventions on a long-term basis can be challenging as they demand considerable resources and people. Our research focuses on investigating the potential use of robotic assistants to allow for these interventions to become more accessible to users and caregivers. Namely, the aim of our work is to develop socially assistive robots that can provide cognitive and social stimulation for persons with dementia. In this paper, we study the social interaction attributes of the human-like robot, Brian 2.0, during a one-on-one person-centered cognitively stimulating activity to determine if the robot is capable of minimizing task-induced stress by providing assistance, encouragement, and celebration, while adapting its behavior to a user state during the course of the activity. Our preliminary study shows that the social intelligence of Brian 2.0 is effective in engaging individuals in a cognitively stimulating game while minimizing stress during gameplay.

This paper presents a novel modular methodology for predicting a lost person's (motion) behavior for autonomous coordinated multirobot wilderness search and rescue. The new concept of isoprobability curves is introduced and developed, which represents a unique mechanism for identifying the target's probable location at any given time within the search area while accounting for influences such as terrain topology, target physiology and psychology, clues found, etc. The isoprobability curves are propagated over time and space. The significant tangible benefit of the proposed target-motion prediction methodology is demonstrated through a comparison to a nonprobabilistic approach, as well as through a simulated realistic wilderness search scenario.

Current applications of mobile robots in urban search and rescue (USAR) environments require a human operator in the loop to help guide the robot remotely. Although human operation can be effective, the unknown cluttered nature of the environments make robot navigation and victim identification highly challenging. Operators can become stressed and fatigued very quickly due to a loss of situational awareness, leading to the robots getting stuck and not being able to find victims in the scene during this time-sensitive operation. In addition, current autonomous robots are not capable of traversing these complex unpredictable environments. To address this challenge, a balance between the level of autonomy of the robot and the amount of human control over the robot needs to be addressed. In this paper, we present a unique control architecture for semi-autonomous navigation of a robotic platform utilizing sensory information provided by a novel real-time 3D mapping sensor. The control system provides the robot with the ability to learn and make decisions regarding which rescue tasks should be carried out at a given time and whether an autonomous robot or a human controlled robot can perform these tasks more efficiently without compromising the safety of the victims, rescue workers and the rescue robot. Preliminary experiments were conducted to evaluate the performance of the proposed collaborative control approach for a USAR robot in an unknown cluttered environment.

The social interaction, guidance and support that a socially assistive robot can provide a person can be very beneficial to patient-centered care. However, there are a number of conundrums that must be addressed in designing such a robot. This work addresses two main limitations in the development of intelligent task-driven socially assistive robots: (i) recognition and identification of human gesticulation as a source of determining the affective state of a person, and (ii) robotic control architecture design and implementation with explicit social and assistive task functionalities. In this paper, the development of a unique task-driven robotic system capable of quantitatively interpreting human body language and in turn, effectively responding via task-driven behavior during assistive social interaction is presented. In particular, a novel gesture identification and classification technique is proposed capable of interpreting human gestures as semantically meaningful commands for inputs into a multi-layer decision making control architecture. The learning-based control architecture is then utilized to determine the effective and appropriate assistive behavior of the robot.

In this paper a unique landmark identification method is proposed for identifying large distinguishable landmarks for 3D visual simultaneous localization and mapping (SLAM) in unknown cluttered urban search and rescue (USAR) environments. The novelty of the method is the utilization of both 3D (i.e., depth images) and 2D images. By utilizing a scale invariant feature transform (SIFT)-based approach and incorporating 3D depth imagery, we can achieve more reliable and robust recognition and matching of landmarks from multiple images for 3D mapping of the environment. Preliminary experiments utilizing the proposed methodology verify: (i) its ability to identify clusters of SIFT keypoints in both 3D and 2D images for representation of potential landmarks in the scene, and (ii) the use of the identified landmarks in constructing a 3D map of unknown cluttered USAR environments.

In this paper the first application of utilizing a unique 3D real-time mapping sensor for sequential 3D map building within a visual simultaneous localization and mapping (SLAM) framework in unknown cluttered urban search and rescue (USAR) environments is proposed. The sensor utilizes a digital fringe projection and phase shifting technique to provide real-time 2D and 3D sensory information of the environment. The proposed sensor is unique over current technologies, in that it can directly map rubble in 3D and in real-time at a frame rate of up to 60 fps. Furthermore, we propose the development of a novel 3D visual SLAM method utilizing both 2D and 3D images taken by the sensor for robust and reliable landmark identification, mapping and localization algorithms utilizing a scale invariant feature transform (SIFT)-based approach. Preliminary experiments show the potential of the proposed 3D real-time sensory system for such unknown cluttered USAR environments.

A novel line-of-sight sensing-based modelless guidance strategy is presented for the autonomous docking of robotic vehicles. The novelty of the proposed guidance strategy is twofold: 1) applicability to situations that do not allow for direct proximity measurement of the vehicle and 2) ability to generate short-range docking motion commands without a need for a global sensing-system (calibration) model. Two guidance -based motion-planning methods were developed to provide the vehicle controller with online corrective motion commands: a passive-sensing-based and an active-sensing-based scheme, respectively. The objective of both proposed guidance methods is to minimize the accumulated systematic errors of the vehicle as a result of the long-range travel, while allowing it to converge to its desired pose within random-noise limits. Both techniques were successfully tested via simulations and experiments, and are discussed herein, in terms of convergence rate and accuracy, in addition to the types of localization problems for which each method could be specifically more suitable.

In this paper a landmark identification method is proposed for identifying large distinguishable landmarks for 3D visual simultaneous localization and mapping (SLAM) in a search and rescue environment. The novelty of the method is the utilization of both 3D (i.e., depth images) and 2D images. By utilizing a scale invariant feature transform (SIFT)-based approach and incorporating 3D depth imagery, we can use more reliable and robust recognition and matching between landmarks from multiple images for 3D mapping of the environment. Preliminary experiments utilizing the proposed method verified its ability to identify clusters of SIFT keypoints in the images for representation of potential landmarks in the scene

In this paper, a neural-network-based guidance methodology that utilizes line-of-sight based task-space sensory feedback is proposed for the localization of autonomous robotic vehicles. The novelty of the overall system is its applicability to cases that do not allow for the direct proximity measurement of the vehicle's pose (position and orientation). Herein, the proposed neural-network (NN) based guidance methodology is implemented on-line during the final stage of the vehicle's motion (i.e., docking). The systematic motion errors of the vehicle are reduced iteratively by executing the corrective motion commands, generated by the NN, until the vehicle achieves its desired pose within random noise limits. The guidance methodology developed was successfully tested via simulations for a 6-dof (degree-of-freedom) vehicle and via experiments for a 3-dof high-precision planar platform

In this paper, an active line-of-sight-sensing (LOS) methodology is proposed for the docking of autonomous vehicles/robotic end-effectors. The novelty of the overall system is its applicability to cases that do not allow for the direct proximity measurement of the vehicle's pose (position and orientation). In such instances, a guidance-based technique must be employed to move the vehicle to its desired pose using corrective actions at the final stages of its motion. The objective of the proposed guidance method is, thus, to successfully minimize the systematic errors of the vehicle, accumulated after a long-range motion, while allowing it to converge within the random noise limits via a three-step procedure: active LOS realignment, determination of the new (actual) location of the vehicle, and implementation of a corrective action. The proposed system was successfully tested via simulation for a three degree-of-freedom (dof) planar robotic platform and via experiments.

This paper addresses the accurate positioning (localization) of a robotic end-effector, undertaking high-precision tasks, by introducing a novel proximity sensing-based point-to-point motion guidance algorithm. The proposed task-space sensing system is only employed at the final stages of motion after the long-range positioning of the end-effector fails to move it to its desired location. Three identical sub-systems that are spatially placed are incorporated into the sensing system, each consisting of a laser emitter, a galvanometer, and a corresponding PSD (position sensitive diode) that is placed directly on the end-effector. The three-step guidance algorithm uses the laser beams and the offsets they produce along the PSDs to guide the end-effector from its actual pose (position and orientation) to its desired pose. The proposed system (sensing and guidance algorithm) was successfully tested via simulation on a three degree-of-freedom (dof) planar parallel manipulator.

In this paper, a multi line-of-sight (LOS) task-space sensing methodology is presented for guidance-based localization of mobile devices (e.g., autonomous vehicles and robots). The mobility requirement of the localization/docking application dictates the minimum number and the type (planar or spatial) of the lines of sight. It is envisioned that, a multi-LOS sensing system will be configured for the task at hand using several, one or two degree-of-freedom (dof), sensing modules. One such module is also proposed in this paper: it comprises a laser source, a (1 or 2 dof) galvanometer mirror and a photodetector. A guidance algorithm would only be invoked at the final stages of vehicle/robotic-end-effector motion after the long-range positioning phase has failed to locate the vehicle at its desired pose (position and orientation). By utilizing a multi-LOS based sensing system the guidance algorithm would successfully minimize the systematic errors of the vehicle, while allowing it to converge to its desired pose within the random noise limits. This has been verified in both simulation and experiments, as presented herein.