Understanding behavior requires datasets that capture humans while carrying out complex tasks. The kitchen is an excellent environment for assessing human motor and cognitive function, as many complex actions are naturally exhibited in kitchens from chopping to cleaning. Here, we introduce the EPFL-Smart-Kitchen-30 dataset, collected in a noninvasive motion capture platform inside a kitchen environment. Nine static RGB-D cameras, inertial measurement units (IMUs) and one head-mounted HoloLens~2 headset were used to capture 3D hand, body, and eye movements. The EPFL-Smart-Kitchen-30 dataset is a multi-view action dataset with synchronized exocentric, egocentric, depth, IMUs, eye gaze, body and hand kinematics spanning 29.7 hours of 16 subjects cooking four different recipes. Action sequences were densely annotated with 33.78 action segments per minute. Leveraging this multi-modal dataset, we propose four benchmarks to advance behavior understanding and modeling through 1) a vision-language benchmark, 2) a semantic text-to-motion generation benchmark, 3) a multi-modal action recognition benchmark, 4) a pose-based action segmentation benchmark. We expect the EPFL-Smart-Kitchen-30 dataset to pave the way for better methods as well as insights to understand the nature of ecologically-valid human behavior. Code and data are available at https://amathislab.github.io/EPFL-Smart-Kitchen

The electromyography (EMG) signal is particularly useful in monitoring muscle activity, and it can be acquired noninvasively on the skin surface. Thanks to these key characteristics, EMG-based human–machine interfaces (HMIs) for prosthetic myocontrol, as well as gesture recognition, are becoming widespread. A key challenge in this context is to design embedded systems to process EMG signals and generate motor commands with miniaturized, unobtrusive, and low-power devices, reliably and in real time, at a relatively low cost to provide continuous monitoring without causing stigma or discomfort. This article presents an in-depth review of the current status and future research challenges in systems and circuits for EMG acquisition and processing. We start by illustrating the sensor interfaces and acquisition systems required for signal analysis to provide efficient and effective ways of understanding the signal and its nature. We, then, focus on conventional state-of-the-art (SoA) EMG gesture recognition algorithms as well as novel architectures that tackle EMG processing challenges, i.e., hyperdimensional computing (HDC), blind source separation (BSS), and spiking neural networks (SNNs). Finally, we discuss open challenges, such as EMG variability, natural control, and efficient computation, to bring the myocontrol completely out of the laboratory, filling the gap between research prototypes and real-world applications.

We developed an extended reciprocity theorem approach to model neuron signals which handles heterogeneous dielectric environments and arbitrary electrode shapes, and use it to study analytically the single fiber action potential. We then established a semi-analytic model that also uses hybrid electromagnetic-electrophysiological simulations to model evoked compound action potential (eCAP) signals from multi-fascicular nerves populated with heterogeneous fiber populations. For validation, model predictions were compared with cuff electrode recordings of activity induced by vagus nerve stimulation in in vivo porcine experiments. The semi-analytic model produces signals that approximate the shape and amplitude of in vivo measurements. It can account for the important variation in the recorded eCAP due to changes in the shape and placement of the stimulus and recording electrodes. We find that partially activated fascicles contribute particularly to the signal, as eCAP contributions from smoothly varying fiber calibers in fully activated ones partially cancel. As a result, eCAP magnitude does not depend monotonically on the stimulation current and recruitment level. Our method can be used to rapidly assess new stimulation and recording setups involving complex nerves and neurovascular bundles, e.g., to maximize signal information content, for closed-loop control in bioelectronic medicine applications, and potentially to non-destructively reconstruct structural and functional nerve topologies through inverse problem solving. In a proof-of-concept study, we demonstrate that parameter optimization can recover the ground-truth distribution of fiber diameters in a simplifed variant of our model.

Recently, the complexity analysis of brain activity has shown the possibility to provide additional information to discriminate between rest and motion in real-time. In this work, we propose a novel entropy-based machine learning method to classify between standing and walking conditions from the sole brain activity. The Shannon entropy has been used as a complexity measure of electroencephalography (EEG) signals and subject-specific features for classification have been selected by Common Spatial Patterns (CSP) filter. Exploiting these features with a linear classifier, we achieved > 85% of classification accuracy over a long period (≈ 25 min) of standing and treadmill walking on 11 healthy subjects. Moreover, we implemented the proposed approach to successfully discriminate in real-time between standing and over-ground walking on one healthy subject. We suggest that the reliable discrimination of rest against walking conditions achieved by the proposed method may be exploited to have more stable control of devices to restore locomotion, avoiding unpredictable and dangerous behaviors due to the delivery of undesired control commands.

We believe that this special issue will serve to increase the public awareness and foster discussions on the multiple worldwide BRAIN initiatives, both within and outside the IEEE, providing an impetus for development of long-term cost-effective healthcare solutions. We also believe that the topics presented in this special issue will serve as scientific evidence for health and policy advocates of the value of neurotechnologies for improving the neurological and mental health and wellbeing of the general population. Below we briefly highlight the papers and technologies in this special issue.

Soft actuators made of highly elastic polymers allow novel robotic system designs, yet application-specific soft robotic systems are rarely reported. Taking notice of the characteristics of soft pneumatic actuators (SPAs) such as high customizability and low inherent stiffness, we report in this work the use of soft pneumatic actuators for a biomedical use - the development of a soft robot for rodents, aimed to provide a physical assistance during gait rehabilitation of a spinalized animal. The design requirements to perform this unconventional task are introduced. Customized soft actuators, soft joints and soft couplings for the robot are presented. Live animal experiment was performed to evaluate and show the potential of SPAs for their use in the current and future biomedical applications.

Medical devices and medical electronics are areas that had little to offer 100 years ago. However, there were three important existing technologies that led to many further developments over the following 100 years. These are the stethoscope, electrocardiography, and X-ray medical imaging. Although these technologies had been described and were available to some extent when the Proceedings of the IEEE pages first appeared, they had yet to achieve the widespread use that they have today. The stethoscope is the oldest of these, and it helped physicians to hear sounds of the body and relate them to functioning and malfunctioning organs. The early use of the stethoscope by physicians was more of an art than a science, but as the Proceedings matured, so did this technology. Engineers were able to make this a more quantitative process by graphically displaying the sounds and ultimately using techniques such as voiceprint analysis to assist the physician in diagnosis and monitoring of treatment. The electrocardiograph had been invented a few years prior to the appearance of the Proceedings, but the apparatus was awkward to use, especially for sick people, and was considered more of an oddity than a viable medical technology 100 years ago. Today, it and devices derived from it such as cardiac patient monitors are important parts of our healthcare system. Similarly, X-rays represented a new technology 100 years ago, but unlike electrocardiography physicians immediately saw the value of this technology and quickly adopted it. Many improvements have been made to the basic technology over the last 100 years culminating in computer tomography and complex image processing. Other devices to create high-quality and 3-D medical images have also been developed in recent years to make medical imaging a very important aspect of clinical care today. Looking to the future is always a difficult task, but it is clear that the electronic health record will play an important role in consolidating the information from various medical devices as well as providing readily available data on patients wherever it might be needed. Future medical devices will need to not only address the problems of diagnostic and therapeutic medicine but also be capable of addressing important societal problems such as worldwide disparities in the availability of medical care, continually rising healthcare costs, and healthcare for travel beyond Earth. The next 100 years promises to be even more exciting than the last from the perspective of medical devices and medical electronics.

The possibility of controlling dexterous hand prostheses by using a direct connection with the nervous system is particularly interesting for the significant improvement of the quality of life of patients, which can derive from this achievement. Among the various approaches, peripheral nerve based intrafascicular electrodes are excellent neural interface candidates, representing an excellent compromise between high selectivity and relatively low invasiveness. Moreover, this approach has undergone preliminary testing in human volunteers and has shown promise. In this paper, we investigate whether the use of intrafascicular electrodes can be used to decode multiple sensory and motor information channels with the aim to develop a finite state algorithm that may be employed to control neuroprostheses and neurocontrolled hand prostheses. The results achieved both in animal and human experiments show that the combination of multiple sites recordings and advanced signal processing techniques (such as wavelet denoising and spike sorting algorithms) can be used to identify both sensory stimuli (in animal models) and motor commands (in a human volunteer). These findings have interesting implications, which should be investigated in future experiments.

An anthropomorphic underactuated prosthetic hand, endowed with position and force sensors and controlled by means of myoelectric commands, is used to perform experiments of hierarchical shared control. Three different hierarchical control strategies combined with a vibrotactile feedback system have been developed and tested by able-bodied subjects through grasping tasks used in activities of daily living (ADLs). The first goal is to find a good tradeoff between good grasping capabilities and low attention required by the user to complete grasping tasks, without addressing advanced algorithm for electromyographic processing. The second goal is to understand whether a vibrotactile feedback system is subjectively or objectively useful and how it changes users' performance. Experiments showed that users were able to successfully operate the device in the three control strategies, and that the grasp success increased with more interactive control. Practice has proven that when too much effort is required, subjects do not do their best, preferring, instead, a less-interactive control strategy. Moreover, the experiments showed that when grasping tasks are performed under visual control, the enhanced proprioception offered by a vibrotactile system is practically not exploited. Nevertheless, in subjective opinion, feedback seems to be quite important.

In recent years, thanks to the advancement of robotics and mechatronics, new and more effective devices for the restoration and replacement of sensory-motor function in disabled people have been developed. In all these systems, user acceptability is strictly connected to several issues such as the residual abilities of the subject, the mechatronic characteristics of the robot, and also the interface chosen to link them. It is possible to figure out different "human-interface-device" combinations [also defined as "hybrid bionic systems" (HBSs)] characterized by different properties in terms of level of hybridness, connection, and augmentation. In particular, in HBSs the interface has to be customized according to the characteristics of the robotic artefact to be controlled and to the desires and needs of the final users. In this paper, our attention has been focused on the problem of the replacement of hand function after amputation. Three HBSs characterized by different levels of complexity, dexterity, and sensorization are presented in order to show the possibility of developing acceptable and effective systems by choosing different levels of connection and hybridness (i.e., different interfaces) for different devices and applications. The following case studies are presented: 1) the use of invasive interfaces to the peripheral nervous system to control a dexterous and highly sensorized hand prosthesis; 2) the use of electromyographic signals recorded using surface electrodes to control a compliant adaptive prosthesis; and 3) the use of a foot interface to control a two-degrees-of-freedom prosthesis. The preliminary results achieved so far seem to confirm the idea that the correct choice of the proper interface while developing an HBS can increase effectiveness and usability

This paper presents recent results aimed at developing a functional prosthetic hand characterized by an EMG-control and by a simple and low cost fabrication technology. In order to overcome some limitations of current prosthetic hands mainly related to the poor functionality and controllability, the prosthetic hand has been designed following a biomechatronic approach based on biologically-inspired design solutions. The core of the project described in this paper is the fabrication of a compliant under-actuated prosthetic hand: the structure of the hand (both palm and fingers) is moulded as a soft polymeric single part with compliant joints and embedded tendon driven underactuated mechanism for providing adaptive grasp. In order to make user trials, the hand is equipped with simple but functional EMG-based control of the single motor incorporated in the hand, and is integrated with a prosthesis socket. The paper presents the biomechatronics design, the fabrication process, the integration of the prosthetic device and first experimental results.

In this paper, some recent results about the experimental trials we are performing on a functional prosthetic hand characterized by an EMG-control and by a simple and low cost fabrication technology are shown. The innovation described in this paper is the fabrication of a compliant under-actuated prosthetic hand. The structure of the hand (both palm and fingers) is moulded as a soft polymeric single part with compliant joints and embedded tendon driven under-actuated mechanism for providing adaptive grasp. The prosthetic hand has been designed following a biomechatronic approach based on biologically-inspired design solutions. We describe the second release of this prosthesis. The evaluating tests are related to the maximum force exerted during grasping and to the fatigue problems occurring with cables and joint.

In this paper a wearable biomechatronic system (named MEKA) for the analysis of knee movements in two degrees of freedom is presented. The system has been designed in order to be used not only in laboratory environment (as many other commercial systems) but also in real-life conditions (e.g. for experiments on sport biomechanics). The performance of the MEKA device and of two commercial systems (an electrogoniometer and a camera-based motion analyzer) has been compared. The results showed that the MEKA system is able to obtain similar or even better performance during movements. After this preliminary assessment phase, the MEKA system has been used to analyse the modifications of motor performance of elderly and young people during gait using a "dual-task" approach. The results show that elderly people seems to be more affected by the increase cognitive efforts. In conclusion, the MEKA system seems to be able to provide interesting information for experiments on biomechanics and motor control.

The aim of this paper was to analyse the feasibility of using a sensor-based glove as a tool for the assessment of hand function in persons with orthopedic disorders such as rhizoarthrosis. A glove embedding 20 Hall-effect sensors was used for this purpose. Its performance in terms of repeatability were preliminary investigated in order to define an effective strategy for the use of the glove in these applications. After this preliminary phase, focused clinical experiments were carried out in order to verify the existence of differences usable for performance assessment in the postures of the hand of able-bodied and subjects affected by rhizoarthrosis during grasping. The data were processed by using the principal component analysis (PCA) technique. The preliminary results of these experiments showed that the use of the first and second principal components can allow to discriminate between able-bodied and disabled persons and also between disabled persons before and after the surgical intervention. For this reason, this approach could represent an effective tool for the surgeon during the rehabilitation process in this particular situation. Further experiments will be carried out in the future in order to confirm these preliminary results. Moreover, a new calibration procedure of the sensor-based glove will be developed in order to improve the performance of the glove.