Expanding the input bandwidth of the human-machine interface to capture more control intentions from the human is key to achieving dexterous control of multi-degree-of-freedom prosthetic hands/wrists. This paper presents a wearable intelligent prosthesis system based on multimodal fusion, which integrates voice interaction, myoelectric control, limb movement decoding, and computer vision-based environmental perception. The system supports intention estimation throughout the entire process from grasping to operation, enabling synchronized control of 4 grasp gestures and 2 wrist DOFs. Moreover, all these decisions are made automatically during the user’s natural and continuous body movements. Six able-bodied subjects and two amputee subjects participated in a comparative experiment involving multi-object grasping and operation in a cluttered environment. Compared to myoelectric pattern recognition, our method demonstrated significant advantages in improving grasp and operation efficiency (reducing total time, grasp time, and operation time by 67.62%, 67.89%, and 67.40% for able-bodied subjects, and by 73.93%, 69.82%, and 76.67% for amputees). It also reduced control burden, with gesture and wrist myocontrol times decreasing by 71.77% and 100% in able-bodied subjects, and by 80.59% and 100% in amputees. These advantages are not limited to the grasp object, grasp part, operation type, or the subject involved. Compared to other semi-autonomous control methods, our method achieved a higher reduction in time metrics, with reduction rates ranging from 22.61% to 84.82% higher, resulting in a more significant performance improvement. Furthermore, the questionnaire showed that the system was well-recognized by the subjects in terms of operational robustness, wearing comfort, and user experience.

Traditional breast cancer surgeries require collaboration between ultrasound (US) doctors and surgeons, making the procedure complex and treating physicians prone to fatigue. In leader-follower robotic surgery, a surgeon controls an US robotic arm and an instrument robotic arm with their left and right hands, enabling independent surgical performance. However, the lack of US scanning skills among surgeons, as well as the physical separation in leader-follower operations, can negatively impact both the scanning and surgical outcomes. This paper proposes a robot-assisted scheme based on dynamic arm impedance compensation (IC) that references expert arm stiffness to compensate for novice arm stiffness. The impedance compensator adjusts the compensation strategy according to the scanning area and scanning stage. The impedance force generator estimates the scanning direction via Kalman filtering and applies stiffness and damping forces in the vertical direction to suppress tremors and other involuntary movements. The experimental results revealed that during the coarse and fine scanning phases, the probe position variance decreased by 57.9% and 73.6%, the contact force variance decreased by 55.2% and 42.5%, and the US image confidence increased by 22.0% and 23.8%, respectively. Compared with traditional filtering compensation (FC) schemes, this approach reduces the average position variance and contact force variance by 32.0% and 25.3%, respectively, and increases confidence by 7.3%. In a no-compensation test, the IC training group outperformed the FC group. This scheme can assist leader-follower US scanning and rapidly improve surgical skills.

Experienced sonographers can adjust their arm impedance in real-time to obtain high-quality ultrasound (US) images during US scanning. These operational skills can be captured through robot systems with multimodal data collection capabilities (position, force, and impedance). However, low operational transparency between the system (generally, a serial robot with admittance control) and its users will result in significant delays and errors, interfering with the skill acquisition process. The paper proposes a new system that adopts the parallel mechanism (Omega.7) to improve the transparency of the operation. The scanning probe and a 6-axis force sensor are attached to the end of Omega.7. When operating the probe, a zero-force drag effect can be realized through gravity and torque compensations. The arm impedance during the scanning can be measured through the force disturbance method by analyzing external forces on the device. Ultrasonic scans were conducted on phantoms of different hardness, and arm impedance was measured. Statistical analysis reveals that when scanning softer phantoms, arms exhibit higher stiffness. The transparency analysis results show that the equipment designed in this paper has a higher level of transparency than the scheme of serial robot with admittance control. The high operation transparency of the system makes it an ideal skill-acquisition device with broad applications.

For a multi-fingered robot hand, the individual control over single joints cannot guarantee their fine collaboration. For achieving a high-precision synchronization, a theory of synchronous control is introduced to multi-fingered robot hands. This paper introduced a new model-free and cross-coupling control strategy. It had been tested on the humanoid robot fingers and showed high positioning performance. For realizing the mutual influence between the control of all joints, we establish the synchronization error by the differential disposal of adjacent actuator errors, then position errors and synchronization errors are incorporated into a unified control frame. Meanwhile, considering the complex dynamic formulations of the dexterous hand and the characteristics of the control system, a model-free, cross-coupled trajectory tracking method is introduced and the explicit dynamic modeling parameters is not necessary. Finally, we tested our method on a multi-fingered hand platform HIT/DLR-II. The results prove that the new method has superior performance over traditional non-synchronous approaches.

Pattern recognition (PR) methods are commonly utilized in the extraction of motion intentions from myoelectric signals, which is realized by relating several electromyogram (EMG) patterns to specific types of motion. Researchers have reported that the hand-engineering features widely used in PR-based methods can be significantly affected by external confounding factors that diminish their accuracy and robustness in clinical settings. Moreover, since only simple mapping is carried out from the feature space to the task space (involving, for the most part, discrete motion intentions), there is no opportunity to exploit fully the underlying mechanism of synergic neuromuscular control. Inspired by deep learning, we have proposed a novel convolutional neural network (CNN) structure based on the characteristics of raw EMG signals that can effectively decode complex wrist movements with three degrees of freedom (DOF) directly from raw EMG signals rather than relying on hand-engineering features. Our method has the potential to incorporate more information than other models by enlarging the training dataset. We demonstrate here that our method performs significantly better (in terms of R2) than the current state-of-art regression method (i.e., support vector regression), especially when confounding factors are involved. We further found that this CNN-based decoding method can be generalized when multiple healthy subjects are taken into account. For a new subject, our method can provide an appropriate control over 3-DOF cursor movements on a screen even without a specific training.

How to design a robotic hand for reproducing the move characteristics of human hand joints is a big challenge in robotics. In this paper, we present an approach to determine the actuation configuration based on the statistical results of hand joint angle in different grasps. A relationship between the basic statistical metrics and actuation configuration strategies is built. In this case, a novel actuation configuration is proposed and the joints of four fingers are arranged into five actuation modules. For the mechanical implementation, the motion of human four finger joints is decomposed to proportion motion, differential motion and chain proportion motion, mechanically implemented by pulley, planetary gear differential module and gear transmission chain. Finally, the implemented mechanism is embedded in palm, and the mechanical implementation to the human hand move characteristics is verified by the measured joint angles of the robotic hand when actuators separately move along PC1 and PC2. Meanwhile, the robotic hand can grasp different objects with a versatile grasp function.

To better restore human hand function, advanced hand prostheses should be able to deal with a variety of daily living conditions. In this paper, we addressed myoelectric signal variations introduced by different muscle contractions, dynamic arm movements, and outer interfering forces in the practice of pattern recognition-based myoelectric control schemes. We examined four different training paradigms (data-collection protocols) and quantified their effectiveness for obtaining a robust classification. We further depicted the classification accuracy according to different arm/wrist motion primitives. Our results indicate the training paradigm that collects myoelectric signals on dynamic arm postures and varying muscular contractions (DPDE) can largely mitigate the motions' misclassification rate. The misclassification rate of finger motions seems to highly correlate to wrist pronation and supination, rather than different arm positions. Combining proprioceptive information, such as the hand's orientation, with myoelectric signals for classification only slightly alleviates the misclassification rate.

Based on HIT/DLR (Harbin Institute of Technology/Deutsches Zentrum für Luft- und Raumfahrt) Prosthetic Hand II, an anthropomorphic controller is developed to help the amputees use and perceive the prosthetic hands more like people with normal physiological hands. The core of the anthropomorphic controller is a hierarchical control system. It is composed of a top controller and a low level controller. The top controller has been designed both to interpret the amputee's intensions through electromyography (EMG) signals recognition and to provide the subject-prosthesis interface control with electro-cutaneous sensory feedback (ESF), while the low level controller is responsible for grasp stability. The control strategies include the EMG control strategy, EMG and ESF closed loop control strategy, and voice control strategy. Through EMG signal recognition, 10 types of hand postures are recognized based on support vector machine (SVM). An anthropomorphic closed loop system is constructed to include the customer, sensory feedback system, EMG control system, and the prosthetic hand, so as to help the amputee perform a more successful EMG grasp. Experimental results suggest that the anthropomorphic controller can be used for multi-posture recognition, and that grasp with ESF is a cognitive dual process with visual and sensory feedback. This process while outperforming the visual feedback process provides the concept of grasp force magnitude during manipulation of objects.

A five-fingered, multi-sensory biomechatronic hand with sEMG interface is presented. The cambered palm is specially designed to enhance the stability while grasping. The location of the thumb is designed by maximizing interaction area between the thumb and other fingers. The opposite thumb could grasp along a cone surface, while maintaining its function. By taken the advantage of coupling linkage mechanism, each finger with three phalanges could fulfill flexion-extension movement independently. Besides, each finger is equipped with torque and position sensors. Thus, the cosmetics and dexterity are improved remarkably compared to conventional prosthesis. The hardware architecture is divided into control system and EMG signal processing system. Moreover, a novel two-stage decision strategy combing the position-based impedance control scheme is implemented to realize the real-time sEMG control of the hand. According to the grasp experiment results, the hand can accomplish several grasp modes stably; the success rate of 10 modes is up to 90%.

The multi-DOF prosthetic hand's myocontrol needs to recognize more hand gestures (or motions) based on myoelectric signals. This paper presents a classification method, which is based on the support vector machine (SVM), to classify 19 different hand gesture modes through electromyographic (EMG) signals acquired from six surface myoelectric electrodes. All hand gestures are based on a 3-DOF configuration, which makes the hand perform like three-fingered. The training performance is very high within each test session, but the cross-session validation is typically low. Acceptable cross-session performance can be achieved by training with more sessions or fewer gesture modes. A fast rhythm muscle contraction is suggested, which can make the training samples more resourceful and improve the prediction accuracy comparing with a relative slow muscle contraction method. For many precise grasp tasks, it is beneficial to the prosthetic hand's myocontrol if we can efficiently extract the grasp force directly from EMG signals. Through grasping a JR3 6 dimension force/torque sensor, the force signal applying to the sensor can be recorded synchronously with myoelectric signals. This paper uses three methods, local weighted projection regression (LWPR), artificial neural network (ANN) and SVM, to find the best regression relationship between these two kinds of signals. It reveals that the SVM method is better than ANN and LWPR, especially in the case of cross-session validation. Also, the performance of grasping force estimation based on specific hand gestures is superior to the performance of grasping with random fingers.