We study goal-conditioned Hierarchical Reinforcement Learning (HRL), where a high-level agent instructs sub-goals to a low-level agent. Under the assumption of a sparse reward function and known hierarchical decomposition, we propose a new algorithm to learn optimal hierarchical policies. Our algorithm takes a low-level policy as input and is flexible enough to work with a wide range of low-level policies. We show that when the algorithm that computes the low-level policy is optimistic and provably efficient, our HRL algorithm enjoys a regret bound which represents a significant improvement compared to previous results for HRL. Importantly, our regret upper bound highlights key characteristics of the hierarchical decomposition that guarantee that our hierarchical algorithm is more efficient than the best monolithic approach. We support our theoretical findings with experiments that underscore that our method consistently outperforms algorithms that ignore the hierarchical structure.

Traditional assessments for dysarthria are subjective and time-consuming, highlighting the need for automated, objective approaches that can be scaled for remote and resource-constrained environments. This paper introduces an adversarial domain adaptation framework tailored for dysarthria severity assessment, addressing the challenges of high intra-class variability and limited availability of dysarthric speech data. By framing speaker variability as a domain adaptation problem, we utilise an adversarially trained feature extractor to derive speaker-invariant yet discriminatively powerful representations utilising speech features learned through self-supervised learning. Experiments on previously unseen and diverse speakers reveal that the proposed approach yields a 7.86% average improvement across multilingual datasets compared to traditional severity-discriminative training, outperforming competitive baselines by 12.33% on average. Additionally, the method inherently supports privacy-preserving applications by minimising reliance on speaker-specific information. The results demonstrate strong alignment with clinical assessments, reinforcing our model’s clinical relevance and effectiveness.

Identifying a brain signal that enables the detection of incorrect execution in human-robot interaction (HRI) is considered a holy grail for real-time systems. A major challenge in achieving this is the inherent imbalance caused by the sparsity of error-related potential (ErrP) events in streaming electroencephalogram (EEG) data, which often leads models to learn irrelevant features and perform poorly. Thus, while deep learning-based ErrP detection has seen considerable advancements, the variability in individual user reaction times introduces labeling errors, complicating model adaptation to new subjects. Moreover, most deep learning methods are developed and validated on discrete, offline experiments using pre-defined windows, which fail to translate effectively to continuous, real-time HRI. Addressing these challenges is crucial to improving the robustness and adaptability of real-time ErrP detection in practical HRI applications. Here, we develop a causal EEG conformer framework, combining a convolutional neural network (CNN) encoder and a transformer with causal attention for real-time prediction of ErrP signals during HRI. We evaluated our ErrP model in a pseudo-online environment in both inter-session and inter-subject cross-validation settings for exoskeleton assistive robotics. Our model demonstrated superior performance in decoding accuracy and efficiency, showcasing better generalization for real-world dynamic HRI applications.

As large language models (LLMs) gain popularity in conducting prediction tasks in-context, understanding the sources of uncertainty in in-context learning becomes essential to ensuring reliability. The recent hypothesis of in-context learning performing predictive Bayesian inference opens the avenue for Bayesian uncertainty estimation, particularly for decomposing uncertainty into epistemic uncertainty due to lack of in-context data and aleatoric uncertainty inherent in the in-context prediction task. However, the decomposition idea remains under-explored due to the intractability of the latent parameter posterior from the underlying Bayesian model. In this work, we introduce a variational uncertainty decomposition framework for in-context learning without explicitly sampling from the latent parameter posterior, by optimising auxiliary inputs as probes to obtain an upper bound to the aleatoric uncertainty of an LLM's in-context learning procedure. Through experiments on synthetic and real-world tasks, we show quantitatively and qualitatively that the decomposed uncertainties obtained from our method exhibit desirable properties of epistemic and aleatoric uncertainty.

Upper limb paralysis affects the quality of life. Functional Electrical Stimulation (FES) offers a solution to restore lost motor functions. Yet, there remain challenges in controlling FES to induce arbitrary arm movements. Reinforcement learning (RL) emerges as a promising method for controlling arm movement with success in simulation. However, challenges remain in translating the successes into real-world settings. One dominant challenge is the sample efficiency of RL. This study presents a practical RL setup to control FES for arm movements. We also present a flexible method, called co-kriging adjustment (CKA), which combines a biomechanical simulator and real data to build an accurate model of the real system. We demonstrate our RL-based control on a 2-DoF planar setting where the subject’s arm, placed on a frictionless supporter, is stimulated to perform point-to-point reaching. By using 90 seconds of real interaction data, our RL-based control can perform the reaching with the average error over the workspace of 5.5 cm. Beyond the application of FES, our method can be extended to other control systems, propelling RL towards general uses in the real world.

We address the challenge of quantifying Bayesian uncertainty and incorporating it in offline use cases of finite-state Markov Decision Processes (MDPs) with unknown dynamics. Our approach provides a principled method to disentangle epistemic and aleatoric uncertainty, and a novel technique to find policies that optimise Bayesian posterior expected value without relying on strong assumptions about the MDP’s posterior distribution. First, we utilise standard Bayesian reinforcement learning methods to capture the posterior uncertainty in MDP parameters based on available data. We then analytically compute the first two moments of the return distribution across posterior samples and apply the law of total variance to disentangle aleatoric and epistemic uncertainties. To find policies that maximise posterior expected value, we leverage the closed-form expression for value as a function of policy. This allows us to propose a stochastic gradient-based approach for solving the problem. We illustrate the uncertainty quantification and Bayesian posterior value optimisation performance of our agent in simple, interpretable gridworlds and validate it through ground-truth evaluations on synthetic MDPs. Finally, we highlight the real-world impact and computational scalability of our method by applying it to the AI Clinician problem, which recommends treatment for patients in intensive care units and has emerged as a key use case of finite-state MDPs with offline data. We discuss the challenges that arise with Bayesian modelling of larger scale MDPs while demonstrating the potential to apply our methods rooted in Bayesian decision theory into the real world. We make our code available at https://github.com/filippovaldettaro/finite-state-mdps.

Proprioception is one of the least understood senses, yet fundamental for the control of movement. Even basic questions of how limb pose is represented in the somatosensory cortex are unclear. We developed a topographic variational autoencoder with lateral connectivity (topo-VAE) to compute a putative cortical map from a large set of natural movement data. Although not fitted to neural data, our model reproduces two sets of observations from monkey centre-out reaching: 1. The shape and velocity dependence of proprioceptive receptive fields in hand-centered coordinates despite the model having no knowledge of arm kinematics or hand coordinate systems. 2. The distribution of neuronal preferred directions (PDs) recorded from multi-electrode arrays. The model makes several testable predictions: 1. Encoding across the cortex has a blob-and-pinwheel-type geometry of PDs. 2. Few neurons will encode just a single joint. Our model provides a principled basis for understanding of sensorimotor representations, and the theoretical basis of neural manifolds, with applications to the restoration of sensory feedback in brain-computer interfaces and the control of humanoid robots.

Hierarchical Reinforcement Learning (HRL) algorithms can perform planning at multiple levels of abstraction. Empirical results have shown that state or temporal abstractions might significantly improve the sample efficiency of algorithms. Yet, we still do not have a complete understanding of the basis of those efficiency gains nor any theoretically grounded design rules. In this paper, we derive a lower bound on the sample complexity for the considered class of goal-conditioned HRL algorithms. The proposed lower bound empowers us to quantify the benefits of hierarchical decomposition and leads to the design of a simple Q-learning-type algorithm that leverages hierarchical decompositions. We empirically validate our theoretical findings by investigating the sample complexity of the proposed hierarchical algorithm on a spectrum of tasks (hierarchical $n$-rooms, Gymnasium's Taxi). The hierarchical $n$-rooms tasks were designed to allow us to dial their complexity over multiple orders of magnitude. Our theory and algorithmic findings provide a step towards answering the foundational question of quantifying the improvement hierarchical decomposition offers over monolithic solutions in reinforcement learning.

Distributional Reinforcement Learning (RL) maintains the entire probability distribution of the reward-to-go, i.e. the return, providing more learning signals that account for the uncertainty associated with policy performance, which may be beneficial for trading off exploration and exploitation and policy learning in general. Previous works in distributional RL focused mainly on computing the state-action-return distributions, here we model the state-return distributions. This enables us to translate successful conventional RL algorithms that are based on state values into distributional RL. We formulate the distributional Bellman operation as an inference-based auto-encoding process that minimises Wasserstein metrics between target/model return distributions. The proposed algorithm, BDPG (Bayesian Distributional Policy Gradients), uses adversarial training in joint-contrastive learning to estimate a variational posterior from the returns. Moreover, we can now interpret the return prediction uncertainty as an information gain, which allows to obtain a new curiosity measure that helps BDPG steer exploration actively and efficiently. We demonstrate in a suite of Atari 2600 games and MuJoCo tasks, including well known hard-exploration challenges, how BDPG learns generally faster and with higher asymptotic performance than reference distributional RL algorithms.

Personalisation is key to creating successful digital health applications. Recent evidence links personality and preference for digital experience — suggesting that psychometric traits can be a promising basis for personalisation of digital mental health services. However, there is still little quantitative evidence from actual app usage. In this study, we explore how different personality types engage with different intervention content in a commercial mental health application. Specifically, we collected the Big Five personality traits alongside the app usage data of 126 participants using a mobile mental health app for seven days. We found that personality traits significantly correlate with the engagement and user ratings of different intervention content. These findings open a promising research avenue that can inform the personalised delivery of digital mental health content and the creation of recommender systems, ultimately improving the effectiveness of mental health interventions.

The intuitive collaboration of humans and intelligent robots (embodied AI) in the real-world is an essential objective for many desirable applications of robotics. Whilst there is much research regarding explicit communication, we focus on how humans and robots interact implicitly, on motor adaptation level. We present a real-world setup of a human-robot collaborative maze game, designed to be non-trivial and only solvable through collaboration, by limiting the actions to rotations of two orthogonal axes, and assigning each axes to one player. This results in neither the human nor the agent being able to solve the game on their own. We use deep reinforcement learning for the control of the robotic agent, and achieve results within 30 minutes of real-world play, without any type of pre-training. We then use this setup to perform systematic experiments on human/agent behaviour and adaptation when co-learning a policy for the collaborative game. We present results on how co-policy learning occurs over time between the human and the robotic agent resulting in each participant's agent serving as a representation of how they would play the game. This allows us to relate a person's success when playing with different agents than their own, by comparing the policy of the agent with that of their own agent.

Assistive robotic systems endeavour to support those with movement disabilities, enabling them to move again and regain functionality. Main issue with these systems is the complexity of their low-level control, and how to translate this to simpler, higher level commands that are easy and intuitive for a human user to interact with. We have created a multi-modal system, consisting of different sensing, decision making and actuating modalities, leading to intuitive, human-in-the-loop assistive robotics. The system takes its cue from the user's gaze, to decode their intentions and implement low-level motion actions to achieve high-level tasks. This results in the user simply having to look at the objects of interest, for the robotic system to assist them in reaching for those objects, grasping them, and using them to interact with other objects. We present our method for 3D gaze estimation, and grammars-based implementation of sequences of action with the robotic system. The 3D gaze estimation is evaluated with 8 subjects, showing an overall accuracy of 4.68\pm 0.14cm. The full system is tested with 5 subjects, showing successful implementation of 100% of reach to gaze point actions and full implementation of pick and place tasks in 96%, and pick and pour tasks in 76% of cases. Finally we present a discussion on our results and what future work is needed to improve the system.

Robotic systems are ever more capable of automation and fulfilment of complex tasks, particularly with reliance on recent advances in intelligent systems, deep learning and artificial intelligence in general. However, as robots and humans come closer together in their interactions, the matter of interpretability, or explainability of robot decision-making processes for the human grows in importance. A successful interaction and collaboration would only be possible through mutual understanding of underlying representations of the environment and the task at hand. This is currently a challenge in deep learning systems. We present a hierarchical deep reinforcement learning system, consisting of a low-level agent handling the large actions/states space of a robotic system efficiently, by following the directives of a high-level agent which is learning the high-level dynamics of the environment and task. This high-level agent forms a representation of the world and task at hand that is interpretable for a human operator. The method, which we call Dot-to-Dot, is tested on a MuJoCo-based model of the Fetch Robotics Manipulator, as well as a Shadow Hand, to test its performance. Results show efficient learning of complex actions/states spaces by the low-level agent, and an interpretable representation of the task and decision-making process learned by the high-level agent.

Recommender Systems have not been explored to a great extent for improving health and subjective wellbeing. Recent advances in mobile technologies and user modelling present the opportunity for delivering such systems, however the key issue is understanding the drivers of subjective wellbeing at an individual level. In this paper we propose a novel approach for deriving personalized activity recommendations to improve subjective wellbeing by maximizing the congruence between activities and personality traits. To evaluate the model, we leveraged a rich dataset collected in a smartphone study, which contains three weeks of daily activity probes, the Big-Five personality questionnaire and subjective wellbeing surveys. We show that the model correctly infers a range of activities that are 'good' or 'bad' (i.e. that are positively or negatively related to subjective wellbeing) for a given user and that the derived recommendations greatly match outcomes in the real-world.

Visual context plays a crucial role in understanding human visual attention in natural, unconstrained tasks - the objects we look at during everyday tasks provide an indicator of our ongoing attention. Collection, interpretation, and study of visual behaviour in unconstrained environments therefore is necessary, however presents many challenges, requiring painstaking hand-coding. Here we demonstrate a proof-of-concept system that enables real-time annotation of objects in an egocentric video stream from head-mounted eye-tracking glasses. We concurrently obtain a live stream of user gaze vectors with respect to their own visual field. Even during dynamic, fast-paced interactions, our system was able to recognise all objects in the user's field-of-view with moderate accuracy. To validate our concept, our system was used to annotate an in-lab breakfast scenario in real time.

Humans process high volumes of visual information to perform everyday tasks. In a reaching task, the brain estimates the distance and position of the object of interest, to reach for it. Having a grasp intention in mind, human eye-movements produce specific relevant patterns. Our Gaze-Contingent Intention Decoding Engine uses eye-movement data and gaze-point position to indicate the hidden intention. We detect the object of interest using deep convolution neural networks and estimate its position in a physical space using 3D gaze vectors. Then we trigger the possible actions from an action grammar database to perform an assistive movement of the robotic arm, improving action performance in physically disabled people. This document is a short report to accompany the Gaze-contingent Intention Decoding Engine demonstrator, providing details of the setup used and results obtained.

We study the problem of off-policy policy evaluation (OPPE) in RL. In contrast to prior work, we consider how to estimate both the individual policy value and average policy value accurately. We draw inspiration from recent work in causal reasoning, and propose a new finite sample generalization error bound for value estimates from MDP models. Using this upper bound as an objective, we develop a learning algorithm of an MDP model with a balanced representation, and show that our approach can yield substantially lower MSE in common synthetic benchmarks and a HIV treatment simulation domain.

Eye movements are closely related to motor actions, and hence can be used to infer motor intentions. Additionally, eye movements are in some cases the only means of communication and interaction with the environment for paralysed and impaired patients with severe motor deficiencies. Despite this, eye-tracking technology still has a very limited use as a human-robot control interface and its applicability is highly restricted to 2D simple tasks that operate on screen based interfaces and do not suffice for natural physical interaction with the environment. We propose that decoding the gaze position in 3D space rather than in 2D results into a much richer “spatial cursor” signal that allows users to perform everyday tasks such as grasping and moving objects via gaze-based robotic teleoperation. Eye tracking in 3D calibration is usually slow - we demonstrate here that by using a full 3D trajectory for system calibration generated by a robotic arm rather than a simple grid of discrete points, gaze calibration in the 3 dimensions can be successfully achieved in short time and with high accuracy. We perform the non-linear regression from eye-image to 3D-end point using Gaussian Process regressors, which allows us to handle uncertainty in end-point estimates gracefully. Our telerobotic system uses a multi-joint robot arm with a gripper and is integrated with our in-house “GT3D” binocular eye tracker. This prototype system has been evaluated and assessed in a test environment with 7 users, yielding gaze-estimation errors of less than 1cm in the horizontal, vertical and depth dimensions, and less than 2cm in the overall 3D Euclidean space. Users reported intuitive, low-cognitive load, control of the system right from their first trial and were straightaway able to simply look at an object and command through a wink to “grasp this” object with the robot gripper.

In this preliminary study, we investigate the potential use of smartphones as portable heart-monitoring devices that can capture and analyse heart activity in real time. We have developed a smartphone application called “Medical Tricorder” that can exploit smartphone;s inertial sensors and when placed on a subject;s chest, it can efficiently capture the motion patterns caused by the mechanical activity of the heart. Using the measured ballistocardiograph signal (BCG), the application can efficiently extract the heart rate in real time while matching the performance of clinical-grade electrocardiographs (ECG). Although the BCG signal can provide much richer information regarding the mechanical aspects of the human heart, we have developed a method of mapping the chest BCG signal into an ECG signal, which can be made directly available to clinicians for diagnostics. Comparing the estimated ECG signal to empirical data from cardiovascular diseases, may allow detection of heart abnormalities at a very early stage without any medical staff involvement. Our method opens up the potential of turning smartphones into portable healthcare systems which can provide patients and general public an easy access to continuous healthcare monitoring. Additionally, given that our solution is mainly software based, it can be deployed on smartphones around the world with minimal costs.

We have deployed body sensor network (BSN) technology in clinical trials and developed behavioural analytics to quantify and monitor longitudinally the progression of Friedreich;s Ataxia (FRDA) outside the lab. Patients and their carers administered themselves our ETHO1 wireless BSN and we captured motion time-series from patient sleep. We extracted behavioural biomarkers that objectively capture the progression of the disease throughout time and compares well with the SARA clinical scale gold-standard. Such clinical scales require patients to go through a series of lengthy tasks where clinicians observe patients' performance and aggregate a score that represents the stage of the disease. Unfortunately, such scales have been shown to be inconsistent across and within clinicians, as they are observation based subjective measures: Scales are highly dependent on the assessor;s experience and they also have low sensitivity and resolution that fails to capture the slow disease progression in short periods of time, requiring longer clinical testing time frames. Using the neurobehavioural data we collected in our clinical trials, we extracted three behavioural biomarkers (MIM, SIM & KIM) based on patient movement intensity, activity and stillness while in bed. Our behavioural biomarkers correlation with the SARA clinical scale allows us to capture the disease progression in FRDA patients and establishes a proof of concept for BSN technology that we are applying towards more rapid efficacy measurements of drugs.

We investigate the effectiveness of a dual-channel MMG signal recorded from the biceps and triceps brachii as a way to predict the isometric forces produced by flexion and extension of the elbow. We asked 8 subjects to apply a range of isometric force levels for both flexion and extension of the elbow while the activity of the two muscles was captured using custom-built MMG sensors. By extracting two characteristic MMG features, the 'MMG score' and the root mean square power spectrum (rmsPS), we applied an artificial feed-forward neural network (NN) to generate a mapping between the MMG signals and the actual forces generated. The accuracy of the NN predictor was evaluated using a 10-fold cross validation, achieving an average across subject R2 of 0.76 and a RMSE of 8.6% of the maximum voluntary isometric contraction (MVC).

Day-long continuous monitoring requires stable sensors that can minimise the effects of drift and maintain high accuracy and precision over time. We have recently shown that our inertial motion tracking system can capture stable kinematic data, calibrated against ground-truth over a long period of time. However, for many clinical and daily life activities, it is also essential to monitor the muscle-activity. In this study, we evaluate the long-term recording stability of our prototype mechanomyography (MMG) sensors as an extension to our existing ETHO1 body sensor network platform. We attached the MMG sensors along with commercial high-accuracy EMG electrodes on the arm muscles of 5 subjects throughout a working day of 9 hours. The subjects followed their daily routine but they had to perform a multi-level force-matching task through flexion and extension of their arm during four short sessions of the day, as measures of practical signal quality. We designed a force predictor that used either EMG or MMG signals to predict the forces generated by subjects. Our prototype low-cost MMG channels have shown comparable results (RMSE: 23N and R2: 0.91) in predicting the force levels applied when compared against the commercial high-accuracy EMG sensor (RMSE: 19N and R2: 0.95).

The vast amounts of data which can be collected using body-sensor networks with high temporal and spatial resolution require a novel analysis approach. In this context, state-of-the-art Bayesian approaches based on variational, non-parametric or MCMC derived methods often become computationally intractable when faced with several million data points. Here, we present how the simple combination of PCA, approximate Bayesian segmentation and temporal correlation processing can achieve reliable time series segmentation. We use our method, which relies on simple iterative covariance, correlation and maximum likelihood operations, to perform complex behavioural time series segmentation over millions of samples in 18 dimensions in linear time and space. Our approach is suitable for even higher dimensional data streams as performance scales near constantly with the dimensionality of the time series samples. We validate this novel approach on an artificially-generated time series and demonstrate that our method is very robust to noise and achieves a segmentation accuracy of over 86% of matching segments against ground-truth. We conclude that our approach makes Big Data driven approaches to stream processing Body Sensor Network (BSN) data tractable, and is required for BSN-driven Neurotechnology applications in Brain-Machine Interfacing and Neuroprosthetics.

Post-synaptic potential (PSP) variability is typically attributed to mechanisms inside synapses, yet recent advances in experimental methods and biophysical understanding have led us to reconsider the role of axons as highly reliable transmission channels. We show that in many thin axons of our brain, the action potential (AP) waveform and thus the Ca++ signal controlling vesicle release at synapses will be significantly affected by the inherent variability of ion channel gating. We investigate how and to what extent fluctuations in the AP waveform explain observed PSP variability. Using both biophysical theory and stochastic simulations of central and peripheral nervous system axons from vertebrates and invertebrates, we show that channel noise in thin axons (<1 µm diameter) causes random fluctuations in AP waveforms. AP height and width, both experimentally characterised parameters of post-synaptic response amplitude, vary e.g. by up to 20 mV and 0.5 ms while a single AP propagates in C-fibre axons. We show how AP height and width variabilities increase with a ¾ power-law as diameter decreases and translate these fluctuations into post-synaptic response variability using biophysical data and models of synaptic transmission. We find for example that for mammalian unmyelinated axons with 0.2 µm diameter (matching cerebellar parallel fibres) axonal noise alone can explain half of the PSP variability in cerebellar synapses. We conclude that axonal variability may have considerable impact on synaptic response variability. Thus, in many experimental frameworks investigating synaptic transmission through paired-cell recordings or extracellular stimulation of presynaptic neurons, causes of variability may have been confounded. We thereby show how bottom-up aggregation of molecular noise sources contributes to our understanding of variability observed at higher levels of biological organisation.

Our hands generic precision and agility is yet unmatched by technology, hence the quantitative study of its daily life kinematics is fundamental to neurology/prosthetics & robotics and creative industries. State-of-the-art solutions capturing hand movements ‘in the wild’ requires wearable body sensor networks: data gloves. Yet, fast-accurate calibration is challenging due to variability in hand anatomy and complexity of finger joints. We present here novel methods for calibration using streaming information from depth cameras (Microsoft Kinect). Our low-cost system calibrates the data glove by observing a user wiggling their hands while wearing data gloves. Using inverse kinematics we reconstruct in real-time hand configuration, enabling augmented reality by superimposing the virtual and real hand veridically. We achieve accuracies of ±5 degrees RMSE over all 21 joints, almost 20% more accurate than standard calibration methods and accurately capture touching of fingertips and thumb — our benchmark test unmatched by other calibration methods.

It is generally assumed that axons use action potentials (APs) to transmit information fast and reliably to synapses. Yet, the reliability of transmission along fibers below 0.5 microm diameter, such as cortical and cerebellar axons, is unknown. Using detailed models of rodent cortical and squid axons and stochastic simulations, we show how conduction along such thin axons is affected by the probabilistic nature of voltage-gated ion channels (channel noise). We identify four distinct effects that corrupt propagating spike trains in thin axons: spikes were added, deleted, jittered, or split into groups depending upon the temporal pattern of spikes. Additional APs may appear spontaneously; however, APs in general seldom fail (<1%). Spike timing is jittered on the order of milliseconds over distances of millimeters, as conduction velocity fluctuates in two ways. First, variability in the number of Na channels opening in the early rising phase of the AP cause propagation speed to fluctuate gradually. Second, a novel mode of AP propagation (stochastic microsaltatory conduction), where the AP leaps ahead toward spontaneously formed clusters of open Na channels, produces random discrete jumps in spike time reliability. The combined effect of these two mechanisms depends on the pattern of spikes. Our results show that axonal variability is a general problem and should be taken into account when considering both neural coding and the reliability of synaptic transmission in densely connected cortical networks, where small synapses are typically innervated by thin axons. In contrast we find that thicker axons above 0.5 microm diameter are reliable.