As Human-Robot Interaction (HRI) and Human-Robot Collaboration (HRC) integrate into society, there's a growing need for flexible, safe, and cost-effective methods to evaluate HRI and HRC from a human perspective. Current evaluation techniques, which often rely on physical hardware, are limited by high costs, safety risks, and low replicability. To address these challenges, we introduce Platform for Evaluating Human-Robot Collaboration and Interaction in Virtual Environments (PEHRCIVE), a software tool built with Unity that leverages Virtual Reality (VR) to provide a safe, flexible, and low-cost research platform. PEHRCIVE provides a customizable, immersive VR environment for HRI and HRC research, based on a collaborative interlocking block assembly task, three distinct collaborative robots (Kuka LBR iiwa, Rethink Robotics Sawyer, and Baxter), and sequential and simultaneous collaboration modes. Features include an easily modifiable external configuration file, a robust data logging system with timestamps and task-specific metrics, and customizable answered-in-VR questionnaires for data collection. PEHRCIVE was validated with 36 participants by collecting physiological and psychological data and responses to the Godspeed Questionnaire Series and VR Presence questionnaire. Participants reported a high sense of presence and a positive VR experience, confirming the platform's effectiveness as a research tool. PEHRCIVE facilitates the design, testing, and evaluation of HRI and HRC experiments more efficiently and safely, accelerating progress for the research community.

Artificial intelligence (AI) and machine learning (ML) algorithms are increasingly being explored to support various decision-making tasks in health (e.g. rehabilitation assessment). However, the development of such AI/ML-based decision support systems is challenging due to the expensive process to collect an annotated dataset. In this paper, we describe the development process of a human-AI collaborative, clinical decision support system that augments an ML model with a rule-based (RB) model from domain experts. We conducted its empirical evaluation in the context of assessing physical stroke rehabilitation with the dataset of three exercises from 15 post-stroke survivors and therapists. Our results bring new insights on the efficient development and annotations of a decision support system: when an annotated dataset is not available initially, the RB model can be used to assess post-stroke survivor’s quality of motion and identify samples with low confidence scores to support efficient annotations for training an ML model. Specifically, our system requires only 22 - 33% of annotations from therapists to train an ML model that achieves equally good performance with an ML model with all annotations from a therapist. Our work discusses the values of a human-AI collaborative approach for effectively collecting an annotated dataset and supporting a complex decision-making task.

Virtual Reality (VR) has become a mainstream opportunity to develop immersive environments where people are exposed to relevant stimuli in a systematic but safe way. Besides the controlled exposure to stimuli, VR systems enable a more efficient and unbiased gathering of information with practical applications in diagnosis, intervention, and health monitoring processes. Nonetheless, scientific literature has demonstrated that interacting with VR technology and its peripherals can be challenging, especially in populations less familiar with technology (e.g., older adults) or with clinical conditions that are not adequately accommodated by the technology, such as cognitive (e.g., dementia) or physical impairments (e.g., blindness). This workshop will focus on systems, technological approaches, and theoretical frameworks to develop user/patient-centred VR tools for diagnosis, intervention, and monitoring of health and wellbeing outcomes. Moreover, we welcome work focused on accessibility issues, aiming to promote equal access and opportunity to use VR tools by people with diverse abilities.

Two of the most popular mediums for virtual reality are head-mounted displays and surround-screen projection systems, such as CAVE Automatic Virtual Environments. In recent years, HMDs suffered a significant reduction in cost and have become widespread consumer products. In contrast, CAVEs are still expensive and remain accessible to a limited number of researchers. This study aims to evaluate both objective and subjective characteristics of a CAVE-like monoscopic low-cost virtual reality surround-screen projection system compared to advanced setups and HMDs. For objective results, we measured the head position estimation accuracy and precision of a low-cost active infrared (IR) based tracking system, used in the proposed low-cost CAVE, relatively to an infrared marker-based tracking system, used in a laboratory-grade CAVE system. For subjective characteristics, we investigated the sense of presence and cybersickness elicited in users during a visual search task outside personal space, beyond arms reach, where the importance of stereo vision is diminished. Thirty participants rated their sense of presence and cybersickness after performing the VR search task with our CAVE-like system and a modern HMD. The tracking showed an accuracy error of 1.66 cm and .4 mm of precision jitter. The system was reported to elicit presence but at a lower level than the HMD, while causing significant lower cybersickness. Our results were compared to a previous study performed with a laboratory-grade CAVE and support that a VR system implemented with low-cost devices could be a viable alternative to laboratory-grade CAVEs for visual search tasks outside the user's personal space.

Advances in artificial intelligence (AI) have made it increasingly applicable to supplement expert’s decision-making in the form of a decision support system on various tasks. For instance, an AI-based system can provide therapists quantitative analysis on patient’s status to improve practices of rehabilitation assessment. However, there is limited knowledge on the potential of these systems. In this paper, we present the development and evaluation of an interactive AI-based system that supports collaborative decision making with therapists for rehabilitation assessment. This system automatically identifies salient features of assessment to generate patient-specific analysis for therapists, and tunes with their feedback. In two evaluations with therapists, we found that our system supports therapists significantly higher agreement on assessment (0.71 average F1-score) than a traditional system without analysis (0.66 average F1-score, p < 0.05). After tuning with therapist’s feedback, our system significantly improves its performance from 0.8377 to 0.9116 average F1-scores (p < 0.01). This work discusses the potential of a human-AI collaborative system to support more accurate decision making while learning from each other’s strengths.

A robotic exercise coaching system requires the capability of automatically assessing a patient's exercise to interact with a patient and generate corrective feedback. However, even if patients have various physical conditions, most prior work on robotic exercise coaching systems has utilized generic, pre-defined feedback.This paper presents an interactive approach that combines machine learning and rule-based models to automatically assess a patient's rehabilitation exercise and tunes with patient's data to generate personalized corrective feedback. To generate feedback when an erroneous motion occurs, our approach applies an ensemble voting method that leverages predictions from multiple frames for frame-level assessment. According to the evaluation with the dataset of three stroke rehabilitation exercises from 15 post-stroke subjects, our interactive approach with an ensemble voting method supports more accurate frame-level assessment (p <; 0.01), but also can be tuned with held-out user's unaffected motions to significantly improve the performance of assessment from 0.7447 to 0.8235 average F1-scores over all exercises (p <; 0.01). This paper discusses the value of an interactive approach with an ensemble voting method for personalized interaction of a robotic exercise coaching system.

Technology-assisted systems to monitor and assess rehabilitation exercises have an opportunity of enhancing rehabilitation practices by automatically collecting patient's quantitative performance data. However, even if a complex algorithm (e.g. Neural Network) is applied, it is still challenging to develop such a system due to patients with various physical conditions. The system with a complex algorithm is limited to be a black-box system that cannot provide explanations on its predictions. To address these challenges, this paper presents a hybrid model that integrates a machine learning (ML) model with a rule-based (RB) model as an explainable artificial intelligence (AI) technique for quantitative assessment of stroke rehabilitation exercises. For evaluation, we collected therapist's knowledge on assessment as 15 rules from interviews with therapists and the dataset of three upper-limb stroke rehabilitation exercises from 15 post-stroke and 11 healthy subjects using a Kinect sensor. Experimental results show that a hybrid model can achieve comparable performance with a ML model using Neural Network, but also provide explanations on a model prediction with a RB model. The results indicate the potential of a hybrid model as an explainable AI technique to support the interpretation of a model and fine-tune a model with user-specific rules for personalization.

Clinical decision support systems have the potential to improve work flows of experts in practice (e.g. therapist's evidence-based rehabilitation assessment). However, the adoption of these systems is challenging, and the gains of these systems have not fully demonstrated yet. In this paper, we identified the needs of therapists to assess patient's functional abilities (e.g. alternative perspectives with quantitative information on patient's exercise motions). As a result, we co-designed and developed an intelligent decision support system that automatically identifies salient features of assessment using reinforcement learning to assess the quality of motion and generate patient-specific analysis. We evaluated this system with seven therapists using the dataset from 15 patients performing three exercises. The results show that therapists have higher usage intent on our system than a traditional system without patient-specific analysis ($p < 0.05$). While presenting richer information ($p < 0.10$), our system significantly reduces therapists' effort on assessment ($p < 0.10$) and improves their agreement on assessment from 0.66 to 0.71 F1-scores ($p < 0.01$). This work discusses the importance of human centered design and development of a machine learning-based decision support system that presents contextually relevant information and salient explanations on its prediction for better adoption in practice.

Due to the limited number of therapists, task-oriented exercises are often prescribed for post-stroke survivors as in-home rehabilitation. During in-home rehabilitation, a patient may become unmotivated or confused to comply prescriptions without the feedback of a therapist. To address this challenge, this paper proposes an automated method that can achieve not only qualitative, but also quantitative assessment of stroke rehabilitation exercises. Specifically, we explored a threshold model that utilizes the outputs of binary classifiers to quantify the correctness of a movements into a performance score. We collected movements of 11 healthy subjects and 15 post-stroke survivors using a Kinect sensor and ground truth scores from primary and secondary therapists. The proposed method achieves the following agreement with the primary therapist: 0.8436, 0.8264, and 0.7976 F1-scores on three task-oriented exercises. Experimental results show that our approach performs equally well or better than multi-class classification, regression, or the evaluation of the secondary therapist. Furthermore, we found a strong correlation (R2 = 0.95) between the sum of computed exercise scores and the Fugl-Meyer Assessment scores, clinically validated motor impairment index of post-stroke survivors. Our results demonstrate a feasibility of automatically assessing stroke rehabilitation exercises with the decent agreement levels and clinical relevance.

The study of neural processes that describe bimanual activity in areas such as neurology and rehabilitation are of high interest, in particular for rehabilitation after brain injury. However, brain processes during bimanual motor rehabilitation are not fully understood during stroke rehabilitation. Hence, it is not clear how to exploit them and their possible advantages in an EEG driven Virtual Reality (VR) training. In this work, VR and EEG were combined to study the neural processes in motor areas during bimanual activity in a serious game, involving two kind of movements: Left to Right (L2R) movement (Right handle forward and Left handle backward movements) and Right to Left (R2L) movement (Right handle backward and Left handle forward movements). 10 right handed healthy people (7 Males, 3 Females, $29.9 \pm 6.21$ years old) participated in this study. As it was expected, differences between rest and bimanual activity conditions (L2R and R2L) were found, surprisingly, on lowest frequency bands, Delta and Theta. More relevant results were found on Delta band at the right Hemisphere and inter-hemispherical relations, specifically for intra-hemispherical connectivity for CPSD relations with p= 0.005 (L2R) and p= 0.02 (R2L), and power quantified with PSD with p= 0.023 (L2R) and p= 0.03 (R2L), while inter-hemispherical connectivity got lower values on resting compared to L2R movement with a p= 0.015. Besides, comparisons between resting and movement in Theta band showed significant results for inter-hemispherical connectivity (p= 0.03, L2R vs Rest, and R2L vs Rest) and differences in power for Left Hemisphere (p= 0.05). Finally, non-significant differences were found in motor cortex between the two kind of bimanual activities tested on this work. These results create an opening scenario to test for mirror effect of bimanual activities from one hemisphere to another on populations with hemi paretic conditions, aiming to apply it in a near future as therapy for Stroke Survivors.

Here, we introduce the design and preliminary validation of a general-purpose architecture for affective-driven procedural content generation in virtual reality (VR) applications in mental health and wellbeing. The architecture supports seven commercial physiological sensing technologies and can be deployed in immersive and non-immersive VR systems. To demonstrate the concept, we developed the "The Emotional Labyrinth," a non-linear scenario in which navigation in a procedurally generated three-dimensional maze is entirely decided by the user, and whose features are dynamically adapted according to a set of emotional states. During navigation, affective states are dynamically represented through pictures, music, and animated visual metaphors chosen to represent and induce affective states. The underlying hypothesis is that exposing users to multimodal representations of their affective states can create a feedback loop that supports emotional self-awareness and fosters more effective emotional regulation strategies. We carried out a first study to, first, assess the effectiveness of the selected metaphors in inducing target emotions, and second, identify relevant psycho-physiological markers of the emotional experience generated by the labyrinth. Results show that the Emotional Labyrinth is overall a pleasant experience in which the proposed procedural content generation can induce distinctive psycho-physiological patterns, generally coherent with the meaning of the metaphors used in the labyrinth design. Furthermore, collected psycho-physiological responses such as electrocardiography, respiration, electrodermal activity, and electromyography are used to generate computational models of users' reported experience. These models enable the future implementation of the closed loop mechanism to adapt the Labyrinth procedurally to the users' affective state.

While CAVE Automatic Virtual Environments (CAVE) have been around for over 2 decades they remain complex to setup, unaffordable to most, and generally limited to data and model visualization applications for academia and industry. In this paper, we present a solution to create a monocular CAVE using the Unity 3D game engine by adding motion parallax and full-body interaction support via the use of a Kinect V2 low-cost sensor. More importantly, we provide a functional and easy to use plugin for that effect, the KAVE, and its configuration tool. Here, we describe our own low-cost CAVE setup, a range of alternative configurations to CAVE systems using this technology and example applications. Finally, we discuss the potential of such an approach considering the current advancements in VR and gaming.

The prospect of connecting the brain and body to a technological device can elicit a broad range of responses from potential users. Early adopters are thrilled by the possibility of a device that can interface directly to the human nervous system. For the vast majority, interest is tempered by caution, as nascent varieties of physiological computing systems raise as many questions as answers about how we will interact with computers in the future.

Worldwide, more than one in three adults suffers from a cardiovascular disease. According to the World Health Organization, 15 million people experience a stroke each year and, of these, 5 million stay permanently disabled. The current limitations of traditional rehabilitation methods push towards the design of personalized tools that can be used intensively by patients and therapists in clinical or at-home environments. In this paper we present the design, implementation and validation of RehabCity, an online game designed for the rehabilitation of cognitive deficits through a gamified approach on activities of daily living (ADLs). Among other findings, our results show a strong correlation between the RehabCity scoring system and the Mini Mental State Examination test for clinical assessment of cognitive function in several domains. These findings suggest that RehabCity is a valid tool for the quantitative assessment of patients with cognitive deficits derived from a brain lesion.

Every year millions of people worldwide suffer from stroke, resulting in motor and/or cognitive disability. As a result, patients experience an increased loss of independence, autonomy and low self-esteem. Evolving to a chronic condition, stroke requires of continuous rehabilitation and therapy. Current ICT approaches, with the use of robotics and Virtual Reality, show some benefits over conventional therapy. However, most of the novel approaches are suitable only for a reduced subset of patients. RehabNet proposes an inclusive approach towards an open and distributed architecture for `in-home' neuro-rehabilitation and monitoring by means of non-invasive ICT. In this paper we present the RehabNet architecture, its design and the implementation of a combined motor-and-cognitive system for post-stroke rehabilitation.

Navigation in unknown dynamic environments still remains a major challenge in robotics. Whereas insects like the desert ant with very limited computing and memory capacities solve this task with great efficiency. Thus, the understanding of the underlying neural mechanisms of insect navigation can inform us on how to build simpler yet robust autonomous robots. Based on recent developments in insect neuroethology and cognitive psychology, we propose a method for landmark navigation in dynamic environments. Our method enables the navigator to learn the reliability of landmarks using an expectation reinforcement method. For that end, we implemented a real-time neuronal model based on the Distributed Adaptive Control framework. The results demonstrate that our model is capable of learning the stability of landmarks by reinforcing its expectations. Also, the proposed mechanism allows the navigator to optimally restore its confidence when its expectations are violated. We also perform navigational experiments with real ants to compare with the results of our model. The behavior of the proposed autonomous navigator closely resembles real ant navigational behavior. Moreover, our model explains navigation in dynamic environments as a memory consolidation process, harnessing expectations and their violations.

Autonomous robotic odour source classification and localization in real world environments is an essential step for applications such as humanitarian demining, environmental monitoring or search and rescue operations. However, at the moment this problem has only been solved by nature (e.g.: moths, bees, rats, dogs). Biological systems are capable and efficient at odour source localization in spite of the difficulties present in the real world such as turbulent environments, obstacles, predators or interfering odours. Here we aim at exploiting our understanding of the moth to solve this problem and we propose a biologically based model of moth behaviour. We implement our model on a robot that uses chemical sensors and we test its performance in a controlled environment. Further, we extend the behavioural model with a sensor front end that supports classification in order to deal with odour distractors. We show that our system is able to locate an odour source and map the chemical environment in the presence of distractors.

Dealing with high-throughput information systems is becoming an everyday problem in many fields of science, as technological advances improve our ability to gather data. In particular, the information encoding problem in high-dimensional spaces is a crucial aspect to consider. In fact, biological systems are known to be very efficient at encoding and processing high-dimensional information. Here we propose a biologically-based solution that mimics the neural processing performed by the Antennal Lobe of insects. Based on our understanding of this system, our model exploits plausible neural mechanisms to transform the massive and high-dimensional spatial and temporal input of the olfactory receptor neurons into a neural population encoding based on synchrony and frequency, consistent with known physiology. We demonstrate the capabilities of our Antennal Lobe model in the context of a classification task of different olfactory stimuli of varying concentrations. We show that the generated neural representation conveys both the identity and the concentration of each stimuli.

In principle it appears advantageous for single neurons to perform non-linear operations. Indeed it has been reported that some neurons show signatures of such operations in their electrophysiological response. A particular case in point is the Lobula Giant Movement Detector (LGMD) neuron of the locust, which is reported to locally perform a functional multiplication. Given the wide ramifications of this suggestion with respect to our understanding of neuronal computations, it is essential that this interpretation of the LGMD as a local multiplication unit is thoroughly tested. Here we evaluate an alternative model that tests the hypothesis that the non-linear responses of the LGMD neuron emerge from the interactions of many neurons in the opto-motor processing structure of the locust. We show, by exposing our model to standard LGMD stimulation protocols, that the properties of the LGMD that were seen as a hallmark of local non-linear operations can be explained as emerging from the dynamics of the pre-synaptic network. Moreover, we demonstrate that these properties strongly depend on the details of the synaptic projections from the medulla to the LGMD. From these observations we deduce a number of testable predictions. To assess the real-time properties of our model we applied it to a high-speed robot. These robot results show that our model of the locust opto-motor system is able to reliably stabilize the movement trajectory of the robot and can robustly support collision avoidance. In addition, these behavioural experiments suggest that the emergent non-linear responses of the LGMD neuron enhance the system's collision detection acuity. We show how all reported properties of this neuron are consistently reproduced by this alternative model, and how they emerge from the overall opto-motor processing structure of the locust. Hence, our results propose an alternative view on neuronal computation that emphasizes the network properties as opposed to the local transformations that can be performed by single neurons.

We present a novel biomimetic approach to mapless autonomous navigation based on insect neuroethology. We implemented and tested a real-time neuronal model based on the Distributed Adaptive Control framework. The model unifies different aspects of insect navigation and foraging including landmark recognition, chemical search, path integration and optimal memory usage. Consistent with recent findings the model supports navigation using heading direction information, thus precluding the use of global information. We tested our model using a mobile robot performing a foraging task. While foraging for chemical sources in a wind tunnel, the robot memorizes the followed trajectories, using information from landmarks and heading direction accumulators. After foraging, landmark navigation is tested with the odor source turned off. Our results show stability against robot kidnapping and generalization of homing behavior to stable mapless landmark navigation. This demonstrates that allocentric and efficient goal-oriented navigation strategies can be generated by relying on purely local information.

Autonomous navigation in 2D and 3D environments has been studied for a long time. Navigating within a 3D environment is very challenging for both animals and robots and a variety of sensors are used to solve this task ranging from vision or a simple gyro or compass to GPS. The principal tasks for 3D autonomous navigation are course stabilization, altitude and drift control, and collision avoidance. Using this basis, some features can be easily added like aerial mapping, object recognition, homing strategies or takeoff and landing. Here we present a biologically based control layer for an Unmanned Aerial Vehicle (UAV) that provides course stabilization, altitude and drift control, and collision avoidance. The properties of this neuronal control system are evaluated using a flying robot.

In insects, we can find very complex and compact neural structures that are task specific. These neural structures allow them to perform complex tasks such as visual navigation, including obstacle avoidance, landing, self-stabilization, etc. Obstacle avoidance is fundamental for successful navigation, and it can be combined with more systems to make up more complex behaviors. In this paper, we present a model for collision avoidance based on the Lobula giant movement detector (LGMD) cell of the locust. This is a wide-field visual neuron that responds to looming stimuli and that can trigger avoidance reactions whenever a rapidly approaching object is detected. Here, we present result based on both an offline study of the model and its application to a flying robot.