For various teleoperation tasks, position-based control is not practical. An impedance-based control is superior e.g. for handling fragile objects, like harvesting fruits or grasping a paper cup. However, only a few researchers have focused on impedance control for teleoperation. In tele-impedance, the stiffness of a human is measured and transferred to a controller of a robot. Until now, human stiffness was mostly measured either for specific joints or in 2D Cartesian space. We introduce a new way of measuring Cartesian stiffness in 3D using electromyography. Users were asked to perform a peg-in-hole task in three different orientations (0°, 45°, 90°). Meanwhile, electromyography measurements at shoulder and elbow muscle groups are performed. In a proof-of-concept study, we showed that the measured stiffness matrix in Cartesian space differed significantly across the three differently oriented peg-in-hole scenarios. This demonstrates that human stiffness could be predicted in 3D Cartesian space based on the type of task at hand.

Neurorobotic devices, such as prostheses, exoskeletons, and muscle stimulators, can partly restore motor functions in individuals with disabilities, such as stroke, spinal cord injury (SCI), and amputations and musculoskeletal impairments. These devices require information transfer from and to the nervous system by neurorobotic interfaces. However, current interfacing systems have limitations of low-spatial and temporal resolution, and lack robustness, with sensitivity to, e.g., fatigue and sensor displacement. Muscle scanning and imaging by ultrasound technology has emerged as a neurorobotic interface alternative to more conventional electrophysiological recordings. While muscle ultrasound detects movement of muscle fibers, and therefore does not directly detect neural information, the muscle fibers are activated by neurons in the spinal cord and therefore their motions mirror the neural code sent from the spinal cord to muscles. In this view, muscle imaging by ultrasound provides information on the neural activation underlying movement intent and execution. Here, we critically review the literature on ultrasound applied as a neurorobotic interface, focusing on technological progresses and current achievements, machine learning algorithms, and applications in both upper-and lower-limb robotics. This critical review reveals that ultrasound in the human-machine interface field has evolved from bulky hardware to miniaturized systems, from multichannel imaging to sparse channel sensing, from simple muscle morphological analysis to input signal for musculoskeletal models and machine learning, from unimodal sensing to multimodal fusion, and from conventional statistical learning to deep learning. For future advances, we recommend exploring high-precision ultrasound imaging technology, improving the wearability and ergonomics of systems and transducers, and developing user-friendly real-time human-machine interaction models.

Emotion labels are usually obtained via either manual annotation, which is tedious and time-consuming, or questionnaires, which neglect the time-varying nature of emotions and depend on human's unreliable introspection. To overcome these limitations, we developed a continuous, real-time, joystick-based emotion annotation framework. To assess the same, 30 subjects each watched 8 emotion-inducing videos. They were asked to indicate their instantaneous emotional state in a valence-arousal (V-A) space, using a joystick. Subsequently, five analyses were undertaken: (i) a System Usability Scale (SUS) questionnaire unveiled the framework's excellent usability; (ii) MANOVA analysis of the mean V-A ratings and (iii) trajectory similarity analyses of the annotations confirmed the successful elicitation of emotions; (iv) Change point analysis of the annotations, revealed a direct mapping between emotional events and annotations, thereby enabling automatic detection of emotionally salient points in the videos; and (v) Support Vector Machines (SVM) were trained on classification of 5 second chunks of annotations as well as their change-points. The classification results confirmed that ratings patterns were cohesive across the participants. These analyses confirm the value, validity, and usability of our annotation framework. They also showcase novel tools for gaining greater insights into the emotional experience of the participants.

The standard paradigm in Affective Computing involves acquiring one/several markers (e.g., physiological signals) of emotions and training models on these to predict emotions. However, due to the internal nature of emotions, labelling/annotation of emotional experience is done manually by humans us ing specially developed annotation tools. To effectively exploit the resulting subjective annotations for developing affective systems, their quality needs to be assessed. This entails, (i) evaluating the variations in annotations, across different subjects and emotional stimuli, to detect spurious/unexpected patterns; and (ii) developing strategies to effectively combine these subjective annotations into a ground truth annotation. This article builds on our previous work by presenting a novel Functional Data Analysis based approach to assess the quality of annotations. Specifically, the bivariate annotation time-series are transformed into functions, such that each resulting functional annotation then becomes a sample element for analysis like Multivariate Functional Principal Component Analysis (MFPCA) that evaluate variation across all annotations. The resulting scores from MFPCA provide interesting insights into annotation patterns and facilitate the use of multivariate statistical techniques to address both (i) and (ii). Given the presented efficacy of these methods, we believe they offer an exciting new approach to assessing the quality of annotations.

Simultaneous and proportional control of hand and wrist prostheses based upon surface electromyography (myocontrol) is still largely an open issue in the community of assistive robotics. It entails the ability of discriminating the activation levels for each degree of freedom (DOF) of the hand/wrist complex, using as few sensors as possible. Furthermore, one should avoid having the human subject train the underlying machine-learning system with all combinations of activations. To tame this problem we have proposed Linearly Enhanced Training (LET), a procedure through which a training set, composed of single-DOF activations provided by the user, is artificially completed with synthetic multi-DOF activations. In this paper, we validate the LET procedure through an online psychophysical experiment carried out on 16 intact subjects and one trans-radial amputee, in which a specific goal must be reached within a determined amount of time. Each subject tried to reach the goal in either of four different scenarios, while the LET procedure was activated or not, and while an optimisation was used or not. A comparative analysis of the results reveals that the usage of LET does not entail any statistically significant difference in the overall performance, and that the usage of the optimisations significantly improves it. Therefore, one can benefit from the drastic reduction of training time due to LET without suffering from significant reduction in performance. The optimisation showed the strong tendency to reduce the time it took to successfully accomplish a task, on average by 1.195s. A comparison of the intact subjects and the trans-radial amputee showed that half of the performance measures of the amputee lie in the 95% confidence interval determined by the able-bodied group.

In the quest for better human-machine interfaces (HMIs) for teleoperation and virtual reality, we hereby present the first integrated application of medical ultrasound imaging to remotely control a virtual piano playing environment in real-time.

Medical ultrasound imaging is a widespread highresolution (both spatial and temporal) method to gather live images of the interior of the human body. Its potential as a human-machine interface for the disabled - amputees in particular - is being explored in the rehabilitation robotics community. Following up the recent discovery that first-order spatial features of the ultrasound images of the human forearm are linearly related to the hand configuration, we hereby push the approach to a realistic scenario. We show that an extremely simple calibration procedure can be used to obtain a linear regression system which will effectively predict the forces required by a human subject at the fingertips, using live ultrasound images of the forearm. In particular, the system can be trained on minimum and maximum forces only, thereby dramatically shortening the calibration phase; and it will generalise to intermediate force values. This phenomenon is uniform across 5 intact subjects whom we examined in a controlled experiment. Moreover, it is not necessary to use any force sensor, as learning-by-imitation, namely using a visual stimulus, yields similar results. This result is particularly useful in the case of amputees, who normally cannot perform graded-force tasks as proprioception may be lost since decades. Applications of this system include, among others: advanced prosthetics, phantom pain therapy and smart teleoperation.

At the time of this writing, the main means of control for polyarticulated self-powered hand prostheses is surface electromyography (sEMG). In the clinical setting, data collected from two electrodes are used to guide the hand movements selecting among a finite number of postures. Machine learning has been applied in the past to the sEMG signal (not in the clinical setting) with interesting results, which provide more insight on how these data could be used to improve prosthetic functionality. Researchers have mainly concentrated so far on increasing the accuracy of sEMG classification and/or regression, but, in general, a finer control implies a longer training period. A desirable characteristic would be to shorten the time needed by a patient to learn how to use the prosthesis. To this aim, we propose here a general method to reuse past experience, in the form of models synthesized from previous subjects, to boost the adaptivity of the prosthesis. Extensive tests on databases recorded from healthy subjects in controlled and noncontrolled conditions reveal that the method significantly improves the results over the baseline nonadaptive case. This promising approach might be employed to pretrain a prosthesis before shipping it to a patient, leading to a shorter training phase.

In this paper we present a novel and simple handheld device for measuring in vivo human grasp impedance. The measurement method is based on a static identification method and intrinsic impedance is identified inbetween 25 ms. Using this device it is possbile to develop continuous grasp impedance measurement methods as it is an active research topic in physiology as well as in robotics, especially since nowadays (bio-inspired) robotics can be impedance-controlled. Potential applications of human impedance estimation range from impedance-controlled telesurgery to limb prosthetics and rehabilitation robotics. We validate the device through a physiological experiment in which the device is used to show a linear relationship between finger stiffness and grip force.

Many real-world tasks require fast planning of highly dynamic movements for their execution in real-time. The success often hinges on quickly finding one of the few plans that can achieve the task at all. A further challenge is to quickly find a plan which optimizes a desired cost. In this paper, we will discuss this problem in the context of catching small flying targets efficiently. This can be formulated as a non-linear optimization problem where the desired trajectory is encoded by an adequate parametric representation. The optimizer generates an energy-optimal trajectory by efficiently using the robot kinematic redundancy while taking into account maximal joint motion, collision avoidance and local minima. To enable the resulting method to work in real-time, examples of the global planner are generalized using nearest neighbour approaches, Support Vector Machines and Gaussian process regression, which are compared in this context. Evaluations indicate that the presented method is highly efficient in complex tasks such as ball-catching.

Ultrasound imaging is a widespread technique to gather live images of the interiors of the human body. It is safe and provides high spatial and temporal resolution. In this paper we show that features extracted from the ultrasound section of the human wrist can be used to fully reconstruct the hand movements, including flexion of all fingers and the rotation of the thumb. Surprisingly, it turns out that there is a clear linear relationship between image features and finger positions. The related matrix can be estimated on a rather small subset of samples, and the reconstruction is quite robust across single- and multi-finger movements. This technique can be used to control advanced mechatronic hands, and it finds its paradigmatic application in the case of hand amputees.

In this paper we describe and practically demonstrate a robotic arm/hand system that is controlled in real-time in 6D Cartesian space through measured human muscular activity. The soft-robotics control architecture of the robotic system ensures safe physical human robot interaction as well as stable behaviour while operating in an unstructured environment. Muscular control is realised via surface electromyography, a non-invasive and simple way to gather human muscular activity from the skin. A standard supervised machine learning system is used to create a map from muscle activity to hand position, orientation and grasping force which then can be evaluated in real time - the existence of such a map is guaranteed by gravity compensation and low-speed movement. No kinematic or dynamic model of the human arm is necessary, which makes the system quickly adaptable to anyone. Numerical validation shows that the system achieves good movement precision. Live evaluation and demonstration of the system during a robotic trade fair is reported and confirms the validity of the approach, which has potential applications in muscle-disorder rehabilitation or in teleoperation where a close-range, safe master/slave interaction is required, and/or when optical/magnetic position tracking cannot be enforced.

The state-of-the-art in control of hand prosthetics is far from optimal. The main control interface is represented by surface electromyography (EMG): the activation potentials of the remnants of large muscles of the stump are used in a non-natural way to control one or, at best, two degrees-of-freedom. This has two drawbacks: first, the dexterity of the prosthesis is limited, leading to poor interaction with the environment; second, the patient undergoes a long training time. As more dexterous hand prostheses are put on the market, the need for a finer and more natural control arises. Machine learning can be employed to this end. A desired feature is that of providing a pre-trained model to the patient, so that a quicker and better interaction can be obtained. To this end we propose model adaptation with least-squares SVMs, a technique that allows the automatic tuning of the degree of adaptation. We test the effectiveness of the approach on a database of EMG signals gathered from human subjects. We show that, when pre-trained models are used, the number of training samples needed to reach a certain performance is reduced, and the overall performance is increased, compared to what would be achieved by starting from scratch.

The dexterity of active hand prosthetics is limited not only due to the limited availability of dexterous prosthetic hands, but mainly due to limitations in interfaces. How is an amputee supposed to command the prosthesis what to do (i.e., how to grasp an object) and with what force (i.e., holding a hammer or grasping an egg)? So far, in literature, the most interesting results have been achieved by applying machine learning to forearm surface electromyography (EMG) to classify finger movements; but this approach lacks, in general, the possibility of quantitatively determining the force applied during the grasping act. In this paper we address the issue by applying machine learning to the problem of regression from the EMG signal to the force a human subject is applying to a force sensor. A detailed comparative analysis among three different machine learning approaches (Neural Networks, Support Vector Machines and Locally Weighted Projection Regression) reveals that the type of grasp can be reconstructed with an average accuracy of 90%, and the applied force can be predicted with an average error of 10%, corresponding to about 5N over a range of 50N. None of the tested approaches clearly outperforms the others, which seems to indicate that machine learning as a whole is a viable approach.

In the framework of indoor mobile robotics, place recognition is a challenging task, where it is crucial that self-localization be enforced precisely, notwithstanding the changing conditions of illumination, objects being shifted around and/or people affecting the appearance of the scene. In this scenario online learning seems the main way out, thanks to the possibility of adapting to changes in a smart and flexible way. Nevertheless, standard machine learning approaches usually suffer when confronted with massive amounts of data and when asked to work online. Online learning requires a high training and testing speed, all the more in place recognition, where a continuous flow of data comes from one or more cameras. In this paper we follow the Support Vector Machines-based approach of Pronobis et al., proposing an improvement that we call Online Independent Support Vector Machines. This technique exploits linear independence in the image feature space to incrementally keep the size of the learning machine remarkably small while retaining the accuracy of a standard machine. Since the training and testing time crucially depend on the size of the machine, this solves the above stated problems. Our experimental results prove the effectiveness of the approach.