Humanlike robotic hands, such as prosthetic hands, become more advanced as technology develops, giving us more lightweight, sophisticated solutions with multiple degrees of freedom. Alongside the hardware improvements, control systems and human machine interfaces are also important areas of research to ensure that the operation of robotic hands is intuitive and easy to master. In particular, amputees are frequently disappointed with the difficulty in controlling their prostheses, which can lead to prostheses rejection. One method that has been explored to reduce the effort and cognitive load on the user is to implement semi-autonomy via appropriate control schemes. In this paper, a semi-autonomous control framework is proposed employing lightmyography based decoding of grasping motions. The proposed framework makes use of video and IMU data so as to reduce the number of possible grasps (grasp affordances) based on the object detected and the hand orientation. The efficiency of the proposed framework has been experimentally validated in comparison to a manual control framework. Using the semi-autonomous framework, misclassifications decreased, leading to 17/20 successful reach to grasps motions executed compared to 7/20 for the manual control case for a single subject. The automatically positioned thumb functionality has also robustified grasping, allowing certain objects to be more dexterously interacted with.

The advancement in robotic grasping and manipulation has elicited an increased research interest in the development of household robots capable of performing a plethora of complex tasks. These advancements require the shift of robotics research from a laboratory setting to dynamic and unstructured home environments. In this work, we focus on a comprehensive data collection and analysis of key attributes involved in the selection of grasping and manipulation strategies for the successful execution of kitchen tasks. An unprecedented dataset that comprises over 7 hours of high-definition videos that were analyzed to classify more than 10,000 kitchen activities annotated with 24 attributes each has been created. Machine learning techniques were employed to automate the annotation process partially by extracting grasp types, hand, and object information from the videos. The annotated dataset was analyzed using clustering algorithms to identify underlying patterns. This study also identifies key attributes and specific data that require focus during data collection based on inter-subject variability. The insights from this study can be used to improve the speed, quality, and effectiveness of data collection. It also helps identify the strategies employed by the humans for the execution of kitchen tasks and transfer the necessary skills to a robotic end-effector enabling it to complete the tasks autonomously or collaborate with humans.

Over the last decades, significant research effort has been put into creating Electromyography (EMG) based controllers for intuitive, hands-free control of robotic arms and hands. To achieve this, machine learning models have been employed to decode human motion and intention using EMG signals as input and to deliver several applications, such as prosthesis control using gesture classification. Despite the advances introduced by new deep learning techniques, real-time control of robot arms and hands using EMG signals as input still lacks accuracy, especially when a plethora of gestures are included as labels in the case of classification. This has been observed to be due to the noise and non-stationarity of the EMG signals and the increased dimensionality of the problem. In this paper, we propose an intuitive, affordances-oriented EMG-based telemanipulation framework for a robot arm-hand system that allows for dexterous control of the device. An external camera is utilized to perform scene understanding and object detection and recognition, providing grasping and manipulation assistance to the user and simplifying control. Object-specific Transformer-based classifiers are employed based on the affordances of the object of interest, reducing the number of possible gesture outputs, dividing and conquering the problem, and resulting in a more robust and accurate gesture decoding system when compared to a single generic classification model. The performance of the proposed system is experimentally validated in a remote telemanipulation setting, where the user successfully performs a set of dexterous manipulation tasks.

Telemanipulation is widely used in robotics applications, ranging from maintenance of various industrial systems to search and rescue response in remote and/or hazardous environments. Human operators are often responsible for the control of such robotic systems. However, these remote interactions require highly trained and experienced operators owing to their complex nature. Semi-autonomous systems are presented as an alternative to complex and counter-intuitive manual systems, combining decoded user intentions with autonomous control modules. This paper proposes a semi-autonomous framework for robotic telemanipulation that employs Electromyography (EMG) based motion decoding and potential fields to execute complex object stacking tasks with a dexterous robot arm-hand system. Even though simple EMG-based teleoperation is promising, the signals are often noisy leading to induced randomness and control errors. To assist the user during task executions, potential fields are utilized to avoid obstacles and guide the robot end-effector toward the objects of interest, thus reducing the cognitive load on the user and the need for accurate predictions. The user's motion is decoded from the myoelectric activations of the human upper arm and upper torso using a Random Forest-based regression methodology. The objects are detected in the environment with an external camera that provides their goal poses to the potential fields scheme. EMG control and potential fields work in a synergistic manner simplifying the system's operation. The framework performance is experimentally validated in real-time experiments involving complex cube and cylinder stacking tasks.

Soft robotic devices have been popular in handling intricate grasping and dexterous manipulation tasks, serving as an alternative to conventional, rigid end-effectors. These devices are relatively simple, lightweight, and cost-effective. Recently, kirigami based structures have been used to create low-cost and disposable soft robotic grippers and hands. These grippers undergo a complex post-contact reconfiguration and conform to an object's shape and size during grasping. In this paper, we explore this new class of soft robotic grippers by utilising them for single-grasp object classification and grasping force estimation. We install simplistic sensors on both the gripper and the actuation system to estimate the state of the kirigami gripper, and the collected data features are employed to train Random Forest models for identifying the grasped object. The classifier trained exhibits a high accuracy of 98 % in discriminating objects of various shapes. When handling food items, the classifier achieves an accuracy of 94 %, while in classifying transparent objects, the classifier obtained again a high accuracy of 97 %. Finally, object-specific force estimation models are triggered based on the classification decision of the Random Forest model to estimate the grasping force exerted by the gripper. These positive outcomes demonstrate the kirigami based robotic gripper's potential for object classification in a variety of circumstances, particularly where vision systems are not available or not reliable.

The advent of collaborative industrial and house-hold robotics has blurred the demarcation between the human and robot workspace. The capability of robots to function efficiently alongside humans requires new research to be conducted in dynamic environments as opposed to the traditional well-structured laboratory. In this work, we propose an efficient skill transfer methodology comprising intuitive interfaces, efficient optical tracking systems, and compliant control of robotic arm-hand systems. The lightweight wearable interfaces mounted with robotic grippers and hands allow the execution of dexterous activities in dynamic environments without restricting human dexterity. The fiducial and reflective markers mounted on the interfaces facilitate the extraction of positional and rotational information allowing efficient trajectory tracking. As the tasks are performed using the mounted grippers and hands, gripper state information can be directly transferred. The hardware-agnostic nature and efficiency of the proposed interfaces and skill transfer methodology are demonstrated through the execution of complex tasks that require increased dexterity, writing and drawing.

For the development of muscle-machine interfaces (MuMIs), researchers have relied mainly on Electromyography (EMG) signals. However, these signals require complex hardware systems, as well as specialized signal processing and feature extraction methods. To overcome these issues, in our previous work, we proposed a novel MuMI for decoding human intention and motion, called Lightmyography (LMG). To improve the performance of this interface even further, in this work, we employ two novel deep learning techniques called Temporal Multi-Channel Transformer (TMC-T) and Temporal Multi-Channel Vision Transformer (TMC-ViT) for the classification of hand gestures based on the LMG data. The performance of these two Transformer-based methods is evaluated and compared with other well-known deep learning and classical machine learning methods. This work also addresses the influence of varying parameters defined during the training phase of decoding models, such as the size and shape of the input data packet. A series of data augmentation techniques were also employed to generate synthetic data and increase the dataset size so as to train deep learning models more efficiently.