User intention estimation is essential in shared control systems for powered wheelchairs. It enables seamless navigation assistance that enhances safety, efficiency, and usability, while preserving user autonomy and reducing effort. This paper presents Diffusion-based Wheelchair User Intention Estimation (DIWIE), a novel multimodal learning framework that leverages a Denoising Diffusion Probabilistic Model (DDPM) to forecast multiple plausible future trajectories, addressing uncertainty in human behaviour. DIWIE conditions on diverse inputs, including obstacle information, user attention cues from eye gaze and head pose, semantic context, wheelchair kinematics, and joystick commands, operating without predefined maps or target destinations. Evaluated on a large new dataset of natural navigation by multiple drivers, DIWIE outperforms state-of-the-art methods, achieving lower displacement errors and collision rates, making it a valuable component for integration into shared control systems. This work also analyses the relevance of different data sources for intention estimation and aligns evaluation metrics with related fields to foster reproducibility.

Despite the growing interest in Behavioural Cloning for robots, few existing research has explicitly explored the impact of user interfaces on the effectiveness of expert demonstrations. We investigate the importance of user interface design in Behavioural Cloning, highlighting the critical role that interfaces play in conveying human demonstrations and robotics capabilities. This study compares the effectiveness of first and third-person perspective interfaces for robot shoe-lacing, a highly dexterous, bi-manual manipulation task that involves deformable objects and requires high precision. Our study highlights the importance of considering the impact of interface design on expert demonstration quality in Behavioural Cloning applications. By providing a first-person perspective, we observed significant differences in demonstration execution time and consistency compared to the third-person perspective. These findings suggest that the choice of interface can influence the quality of expert demonstrations, which in turn affects the performance of learning algorithms.

Assistance through wheelchair-mounted manipulators has the potential to enhance the independence of individuals with disabilities. However, existing approaches primarily focus on single-arm systems or require extensive user input and demonstrations to infer intentions. In this study, we present a shared control framework for intuitive dual-arm operation through a standard 2D joystick, focusing on pick-and-place tasks. Our approach infers user intent in real-time, eliminating the need for specifying prior goals or beliefs. To address the challenges of controlling two 7-DoF (Kinova Gen3) manipulators, we propose two distinct control methods: one for pre-grasp positioning and one for grasp execution. The first method employs a shared control policy to optimize the pre-grasp positioning of the wheelchair base, ensuring ergonomic alignment for front-grasping tasks while incorporating mobile manipulation to reduce task completion time. The second method allows users to maintain high-level goal control and fine-tuning through task-specific arbitration. Experimental results demonstrate high grasp quality and task efficiency across pick-and-place scenarios, establishing the feasibility of shared control for bimanual manipulation and wheelchair navigation. We believe this is the first unified framework for mobile bimanual manipulation using standard wheelchair controls.

We introduce a comprehensive approach to examining the complexities of trust during Human–Robot Interactions (HRIs) through an innovative 3D eye-gaze tracking framework. Trust is a fundamental psychological factor in HRI studies, influencing how humans perceive and interact with robots. Although researchers have previously highlighted eye-tracking as a promising tool for capturing behavioural manifestations of trust continuously and non-intrusively, traditional approaches have been limited to 2D setups, leaving their applicability to real-world HRI largely unexplored. Thus, there still is limited evidence for the feasibility and validity of using eye-tracking to assess human–robot trust in more realistic settings. To this end, our framework employs Head-Mounted Displays with 3D eye-gaze and spatial tracking capabilities to gather continuous eye-gaze data alongside real-time user and robot positions. In addition to 3D eye-gaze tracking capabilities, we designed and incorporated a Bayesian model to evaluate experimental treatments’ effectiveness while identifying eye-gaze features correlating with participants’ subjective trust scores. The latter are measured using Likert-type instruments, widely used in HRI research. We applied our framework to a user study involving 25 participants performing an inspection task with a robot under two reliability conditions—high versus low. Our results revealed significant differences in subjective trust between conditions. Moreover, the results show that participants exposed to the low-reliability condition fixate for longer and have higher fixation and saccade amplitudes when compared to those in the high-reliability condition. Additionally, the group with low reliability had a greater rate of transitions between fixations. These findings are consistent with previous research on 2D settings. However, we observed differences in scan-path length and total fixation count compared to previous studies. Lastly, our results show that incorporating multiple eye-gaze feature categories simultaneously into our Bayesian model can lead to a more nuanced comprehension of the intricate connections between eye-gaze patterns and subjective trust in HRI. A supplementary video providing additional details is available online as supplementary material and can also be accessed at https://www.imperial.ac.uk/personal-robotics/videos/ .

Augmented Reality multi-user interfaces facilitate communication, coordination and collaboration among teams. Moreover, these interfaces can help to align the team’s perceptions and expectations under a shared mental model. This model is a psychological construct that represents the common knowledge, beliefs, and understandings held by team members. In this paper, we study to what extent, if any, the combination of Augmented Reality multi-user interfaces and shared mental models affects human-robot trust. To this end, we developed an Augmented Reality multi-user interface to perform a user study (N = 37) comparing non-dyadic human-robot interactions with a quadruped robot exhibiting low reliability (Group 3), against dyadic interactions while the robot exhibited high-reliability (Group 1) or low-reliability (Group 2). We made this comparison using validated trust questionnaires relevant to HRI. Our results, obtained via Bayesian data analysis methods, show differences in the distribution of answers between groups 1 and 2. Notably, this difference is smaller between groups 1 and 3, which suggests that the combination of shared mental models and multi-user interfaces holds promise as an effective way to manage and calibrate human-robot trust.

Abstract Federated learning is the state-of-the-art paradigm for training a learning model collaboratively across multiple distributed devices while ensuring data privacy. Under this framework, different algorithms have been developed in recent years and have been successfully applied to real use cases. The vast majority of work in federated learning assumes static datasets and relies on the use of deep neural networks. However, in real-world problems, it is common to have a continual data stream, which may be non-stationary, leading to phenomena such as concept drift. Besides, there are many multi-device applications where other, non-deep strategies are more suitable, due to their simplicity, explainability, or generalizability, among other reasons. In this paper we present Ensemble and Continual Federated Learning, a federated architecture based on ensemble techniques for solving continual classification tasks. We propose the global federated model to be an ensemble, consisting of several independent learners, which are locally trained. Thus, we enable a flexible aggregation of heterogeneous client models, which may differ in size, structure, or even algorithmic family. This ensemble-based approach, together with drift detection and adaptation mechanisms, also allows for continual adaptation in situations where data distribution changes over time. In order to test our proposal and illustrate how it works, we have evaluated it in different tasks related to human activity recognition using smartphones.

Learning from Demonstration (LfD) is a very appealing approach to empower robots with autonomy. Given some demonstrations provided by a human teacher, the robot can learn a policy to solve the task without explicit programming. A promising use case is to endow smart robotic wheelchairs with active assistance to navigation. By using LfD, it is possible to learn to infer short-term destinations anywhere, without the need of building a map of the environment beforehand. Nevertheless, it is difficult to generalize robot behaviors to environments other than those used for training. We believe that one possible solution is learning from crowds, involving a broad number of teachers (the end users themselves) who perform demonstrations in diverse and real environments. To this end, in this work we consider Federated Learning from Demonstration (FLfD), a distributed approach based on a Federated Learning architecture. Our proposal allows the training of a global deep neural network using sensitive local data (images and laser readings) with privacy guarantees. In our experiments we pose a scenario involving different clients working in heterogeneous domains. We show that the federated model is able to generalize and deal with non Independent and Identically Distributed (non-IID) data.

Abstract Smart devices, such as smartphones, wearables, robots, and others, can collect vast amounts of data from their environment. This data is suitable for training machine learning models, which can significantly improve their behavior, and therefore, the user experience. Federated learning is a young and popular framework that allows multiple distributed devices to train deep learning models collaboratively while preserving data privacy. Nevertheless, this approach may not be optimal for scenarios where data distribution is non-identical among the participants or changes over time, causing what is known asconcept drift. Little research has yet been done in this field, but this kind of situation is quite frequent in real life and poses new challenges to both continual and federated learning. Therefore, in this work, we present a new method, called Concept-Drift-Aware Federated Averaging (CDA-FedAvg). Our proposal is an extension of the most popular federated algorithm, Federated Averaging (FedAvg), enhancing it for continual adaptation under concept drift. We empirically demonstrate the weaknesses of regular FedAvg and prove that CDA-FedAvg outperforms it in this type of scenario.

Artificial intelligence and all its supporting tools, e.g. machine and deep learning in computational intelligence-based systems, are rebuilding our society (economy, education, life-style, etc.) and promising a new era for the social welfare state. In this paper we summarize recent advances in data science and artificial intelligence within the interplay between natural and artificial computation. A review of recent works published in the latter field and the state the art are summarized in a comprehensive and self-contained way to provide a baseline framework for the international community in artificial intelligence. Moreover, this paper aims to provide a complete analysis and some relevant discussions of the current trends and insights within several theoretical and application fields covered in the essay, from theoretical models in artificial intelligence and machine learning to the most prospective applications in robotics, neuroscience, brain computer interfaces, medicine and society, in general.