One of the most important research challenges in upper-limb prosthetics is enhancing the user-prosthesis communication to closely resemble the experience of a natural limb. As prosthetic devices become more complex, users often struggle to control the additional degrees of freedom. In this context, leveraging shared-autonomy principles can significantly improve the usability of these systems. In this paper, we present a novel eye-in-hand prosthetic grasping system that follows these principles. Our system initiates the approach-to-grasp action based on user's command and automatically configures the DoFs of a prosthetic hand. First, it reconstructs the 3D geometry of the target object without the need of a depth camera. Then, it tracks the hand motion during the approach-to-grasp action and finally selects a candidate grasp configuration according to user's intentions. We deploy our system on the Hannes prosthetic hand and test it on able-bodied subjects and amputees to validate its effectiveness. We compare it with a multi-DoF prosthetic control baseline and find that our method enables faster grasps, while simplifying the user experience. Code and demo videos are available online at this https URL.

Most control techniques for prosthetic grasping focus on dexterous fingers control, but overlook the wrist motion. This forces the user to perform compensatory movements with the elbow, shoulder and hip to adapt the wrist for grasping. We propose a computer vision-based system that leverages the collaboration between the user and an automatic system in a shared autonomy framework, to perform continuous control of the wrist degrees of freedom in a prosthetic arm, promoting a more natural approach-to-grasp motion. Our pipeline allows to seamlessly control the prosthetic wrist to follow the target object and finally orient it for grasping according to the user intent. We assess the effectiveness of each system component through quantitative analysis and finally deploy our method on the Hannes prosthetic arm. Code and videos: https: //hsp-iit.github.io/hannes-wrist-control.

Recent advancements in control of prosthetic hands have focused on increasing autonomy through the use of cameras and other sensory inputs. These systems aim to reduce the cognitive load on the user by automatically controlling certain degrees of freedom. In robotics, imitation learning has emerged as a promising approach for learning grasping and complex manipulation tasks while simplifying data collection. Its application to the control of prosthetic hands remains, however, largely unexplored. Bridging this gap could enhance dexterity restoration and enable prosthetic devices to operate in more unconstrained scenarios, where tasks are learned from demonstrations rather than relying on manually annotated sequences. To this end, we present HannesImitationPolicy, an imitation learning-based method to control the Hannes prosthetic hand, enabling object grasping in unstructured environments. Moreover, we introduce the HannesImitationDataset comprising grasping demonstrations in table, shelf, and human-to-prosthesis handover scenarios. We leverage such data to train a single diffusion policy and deploy it on the prosthetic hand to predict the wrist orientation and hand closure for grasping. Experimental evaluation demonstrates successful grasps across diverse objects and conditions. Finally, we show that the policy outperforms a segmentation-based visual servo controller in unstructured scenarios. Additional material is provided on our project page: https://hsp-iit.github.io/HannesImitation.

Although many advancements have been made on myoelectric pattern-recognition, the control of poly-articulated upper-limb prostheses remains insufficiently robust. Electrode-shift, sweat or fatigue degrade the performance of classifiers over time, resulting in unfruitful device usage and frequent re-calibration. To tackle this issue, here we introduce two models − µP6 and µP8 − that combine Geometric Algebra with nearest-neighbor classification. We aim at reducing both the necessary training data and training time and, unlike most current state-of-the-art algorithms, we exploit an alternative geometric representation (and visualization) of the EMG signal as different polygons for different types of gestures, facilitating the explanation of the decision-making process to a layman. Moreover, we explore four abstention strategies to reduce the number of misclassifications. We perform an offline analysis on two datasets, alongside two other standard models: nonlinear logistic regression (NLR) and linear discriminant analysis (LDA). Even with few training data, the proposed algorithms achieve high F1-scores (>0.95), significantly higher or non-significantly different from the values obtained with NLR and LDA, while maintaining relatively low abstentions rates and training times (<2 ms). The proposed algorithms allow to reduce the amount of training data and training times without compromising recognition rates. The proposed algorithms may contribute for a faster prosthesis re-calibration procedure while allowing to re-gain high recognition rates. Furthermore, the decision-making process is explainable and interpretable, potentially improving user trust and acceptance.

Stability and repeatability of Pattern Recognition (PR) myoelectric control for upper limb prosthetic devices remain unresolved challenges in multi-DoFs systems. In this study, we tested several state-of-the-art classifiers to compare their offline performance in different configurations. Parameters such as realization costs, overall encumbrance, and algorithm complexity were considered for the analysis. The results showed that NLR performed comparably to LDA but with fewer EMG sensors. This study demonstrated that sensor numbers can be reduced to a few units for various algorithms, with NLR being the most tolerant due to its non-linearity. In conclusion, NLR can effectively control the multi-DoFs Hannes system in real-time, offering similar performances to other algorithms while reducing the system's complexity and encumbrance as compared to LDA. It also offers improved tolerance to reduced available information and lower implementation costs.

5G is being deployed and B5G connectivity is under study and standardization. Benefits brought by the 5G/B5G air interface are numerous and 5G is more than just an evolution of radio technology since it consists of innovative concepts: the application of network softwarization and programmability paradigms to the overall network design, the reduced latency promised by edge computing, or the concept of network slicing. These innovations open the door to new vertical-specific services, even capable of saving more lives. The paper describes four use cases to demonstrate the large-scale trialing of the B5G technology specifically devoted to eHealth and Emergency domains, by supporting the B5G applications in large-scale environments (e.g., hospitals) and bringing novel applications (e.g., Remote Proctoring and Smart Ambulance) and on societal benefits in eHealth and Emergency areas through the development of innovative B5G/6G applications. The work is a part of a more complete behavior, TrialsNet project, within SNS JU European Commission Programme, considering other field of application of B5G connectivity.

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-in-hand systems automatically control the hand pre-shaping before the grasp, based on visual input coming from a camera on the wrist. In this paper, we present an eye-in-hand learning-based approach for hand pre-shape classification from RGB sequences. Differently from previous work, we design the system to support the possibility to grasp each considered object part with a different grasp type. In order to overcome the lack of data of this kind and reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. We make publicly available the code and dataset to reproduce the presented results11https://github.com/hsp-iit/prosthetic-grasping-simulation.

Recent literature suggests that self-regulation techniques like biofeedback can be used to enhance the embodiment of artificial limbs. In this study, we developed and preliminarily tested an embodiment training protocol based on a Spatial Augmented Respiratory Cardiofeedback (SARC) implemented through a computer screen - visualizing a 3D model of a prosthetic hand (Hannes) - and a thoracic band for monitoring the Heart Rate Variability (HRV) of the users. The feedback was based on the respiratory-driven modulation of a composite index of the individuals’ cardiac autonomic state after an initial calibration based on slow breathing (at a rate perceived as “comfortable”). Alongside the assessment of the SARC use feasibility, this pilot study evaluates the virtual hand embodiment obtained in two task conditions. In both conditions, the virtual limb gradually appears when the cardiofeedback exercise is performed correctly. Otherwise, the virtual limb parts gradually disappear (“unstable” condition) or they remain visible (cumulative” condition). In the latter case, the virtual hand maintains its “reality-based” stability, supporting the subject’s motivation. Ten volunteers without disabilities were presented both conditions on 10 trials each (2min per trial). Their experience and their proprioceptive drift (estimating their real hand position as close to the artificial one) were assessed as measures of virtual prosthesis embodiment. The questionnaire results preliminarily highlight the feasibility of the SARC. Furthermore, a significantly stronger drift for the virtual prosthesis occurred in the cumulative condition, orienting further investigations.

The quality of life for upper limb amputees can be greatly improved by the adoption of poly-articulated myoelectric prostheses. Typically, in these applications, a pattern recognition algorithm is used to control the system by converting the recorded electromyographic activity (EMG) into complex multi-degrees of freedom (DoFs) movements. However, there is currently a trade-off between the intuitiveness of the control and the number of active DoFs. We here address this challenge by performing simultaneous multi-joint control of the Hannes system and testing several state-of-the-art classifiers to decode hand and wrist movements. The algorithms discriminated multi-DoF movements from forearm EMG signals of 10 healthy subjects reproducing hand opening-closing, wrist flexion-extension and wrist pronation-supination. We first explored the effect of the number of employed EMG electrodes on device performance through the classifiers optimization in terms of F1Score. We further improved classifiers by tuning their respective hyperparameters in terms of the Embedding Optimization Factor. Finally, three mono-lateral amputees tested the optimized algorithms to intuitively and simultaneously control the Hannes system. We found that the algorithms performances were similar to that of healthy subjects, particularly identifying the Non-Linear Regression classifier as the ideal candidate for prosthetic applications.