Robots have predominantly been used in automating tasks in structured industrial environments, however, with the advances in technology they are starting to take part in roles in dynamic everyday life scenarios. As a result, the tasks executed by robotic systems will also grow in sophistication. Grasping and dexterous manipulation are critical aspects that allow humans to execute these sophisticated tasks, enabling them to interact with their environment. As such, emulating the human hand can be advantageous for interacting with a world designed for humans. However, directly replicating the anatomical structure of the hand produces designs that are fully actuated, expensive, and which require sophisticated controls and sensing to operate efficiently. In this paper, we present two different versions of the New Dexterity adaptive, humanlike robot hand that is capable of executing robust caging grasps under a wide range of environmental uncertainties (e.g., object pose uncertainties). One of the versions has a classic, fixed thumb base while the second one incorporates an additional degree of freedom at the thumb base, which enables a translational motion for repositioning the thumb and adjusting the aperture. This design choice enhances the inhand manipulation capabilities of the robot hand, improving also the power grasping capabilities for larger objects. The performances of the proposed robot hand designs are experimentally validated and compared through three different tests: i) grasping experiments involving everyday-life objects, ii) force experiments that evaluate their force exertion capabilities, and iii) in-hand manipulation experiments that demonstrate and compare their dexterity.

Over the last decades, significant research effort has been put into creating Electromyography (EMG) based controllers for intuitive, hands-free control of robotic arms and hands. To achieve this, machine learning models have been employed to decode human motion and intention using EMG signals as input and to deliver several applications, such as prosthesis control using gesture classification. Despite the advances introduced by new deep learning techniques, real-time control of robot arms and hands using EMG signals as input still lacks accuracy, especially when a plethora of gestures are included as labels in the case of classification. This has been observed to be due to the noise and non-stationarity of the EMG signals and the increased dimensionality of the problem. In this paper, we propose an intuitive, affordances-oriented EMG-based telemanipulation framework for a robot arm-hand system that allows for dexterous control of the device. An external camera is utilized to perform scene understanding and object detection and recognition, providing grasping and manipulation assistance to the user and simplifying control. Object-specific Transformer-based classifiers are employed based on the affordances of the object of interest, reducing the number of possible gesture outputs, dividing and conquering the problem, and resulting in a more robust and accurate gesture decoding system when compared to a single generic classification model. The performance of the proposed system is experimentally validated in a remote telemanipulation setting, where the user successfully performs a set of dexterous manipulation tasks.

Telemanipulation is widely used in robotics applications, ranging from maintenance of various industrial systems to search and rescue response in remote and/or hazardous environments. Human operators are often responsible for the control of such robotic systems. However, these remote interactions require highly trained and experienced operators owing to their complex nature. Semi-autonomous systems are presented as an alternative to complex and counter-intuitive manual systems, combining decoded user intentions with autonomous control modules. This paper proposes a semi-autonomous framework for robotic telemanipulation that employs Electromyography (EMG) based motion decoding and potential fields to execute complex object stacking tasks with a dexterous robot arm-hand system. Even though simple EMG-based teleoperation is promising, the signals are often noisy leading to induced randomness and control errors. To assist the user during task executions, potential fields are utilized to avoid obstacles and guide the robot end-effector toward the objects of interest, thus reducing the cognitive load on the user and the need for accurate predictions. The user's motion is decoded from the myoelectric activations of the human upper arm and upper torso using a Random Forest-based regression methodology. The objects are detected in the environment with an external camera that provides their goal poses to the potential fields scheme. EMG control and potential fields work in a synergistic manner simplifying the system's operation. The framework performance is experimentally validated in real-time experiments involving complex cube and cylinder stacking tasks.

For the development of muscle-machine interfaces (MuMIs), researchers have relied mainly on Electromyography (EMG) signals. However, these signals require complex hardware systems, as well as specialized signal processing and feature extraction methods. To overcome these issues, in our previous work, we proposed a novel MuMI for decoding human intention and motion, called Lightmyography (LMG). To improve the performance of this interface even further, in this work, we employ two novel deep learning techniques called Temporal Multi-Channel Transformer (TMC-T) and Temporal Multi-Channel Vision Transformer (TMC-ViT) for the classification of hand gestures based on the LMG data. The performance of these two Transformer-based methods is evaluated and compared with other well-known deep learning and classical machine learning methods. This work also addresses the influence of varying parameters defined during the training phase of decoding models, such as the size and shape of the input data packet. A series of data augmentation techniques were also employed to generate synthetic data and increase the dataset size so as to train deep learning models more efficiently.

Flexible flat cables (FFC) are very popular for connecting different components in modern electronics (e.g., mobile phones, laptops, tablets, etc.). The manipulation of FFCs typically relies on highly trained workers that spend hours performing the same repetitive processes, or on autonomous robotic systems that are equipped with simple clamping mechanisms or pneumatically driven suction cups. Such robotic systems are difficult to program and reprogram and often rely on sophisticated sensing elements and complicated control laws. Moreover, the performance and robustness of such systems is far from sufficient, hindering their mass adoption. The manipulation of FFCs is also quite challenging. A good gripper should be able to pinch the cable steadily and execute insertion tasks of the cable connector with ease. The suction cup based solution is a good approach for holding the cable, but it makes the cable connector insertion very challenging as it can only apply limited shear forces. In this paper, we propose a locally-adaptive, pneumatic, parallel-jaw robot gripper equipped with fingertips that are able to both pinch the cable with a soft clamping mechanism and roll the cable surface on the soft fingertip structure until it reaches the desired connector. The gripper base accommodates a camera that allows for the recognition and pose estimation of the flat, flexible cables and other electronic components. The gripper is of low-cost and low-complexity and it can facilitate the efficient and robust execution of FFC grasping and assembly tasks.

Electromyography (EMG) is a wearable, noninvasive, commonly used method for measuring the human muscular activations from the surface of the skin. In this work, we present a pilot study that focuses on the formulation of a shared control framework to facilitate the simplified execution of Electromyography (EMG) based telemanipulation tasks with a robotic platform. The framework combines a Random Forests (RF) regression method with a compliance controller that relies on the force measurements collected with a force-torque sensor. The RF regression efficiently maps the myoelectric activations of the human muscles to corresponding human wrist positions. Then, a teleoperation process is used to control the robot arm end-effector’s position, utilizing the human wrist position estimations. The examined application involves semi-autonomous cleaning of a whiteboard surface with the proposed framework. The compliance controller guarantees that a desired contact force will always be maintained on the whiteboard surface during task execution. This ensures that any EMG based decoding inaccuracies will not drive the robot away from the cleaning plane. Essentially, the system projects the EMG based estimation on the cleaning plane. The shared control framework offers robust performance, with minimal training and calibration required.

With an increasing number of robotic and prosthetic devices, there is a need for intuitive interfaces which enable the user to efficiently interact with them. The conventional interfaces are generally bulky and unsuitable for dynamic and unstructured environments. An alternative to the traditional interfaces is the class of Muscle-Machine Interfaces (MuMIs) that allow the user to have an embodied interaction with the devices they are controlling. In this work, we present a wearable, lightweight, Forcemyography (FMG) based armband for Human-Machine Interaction fabricated entirely out of 3D-printed parts and silicone components. The armband uses six force sensing units, each housing an Force Sensitive Resistor (FSR) sensor. The capabilities of the armband are evaluated while decoding four different gestures (pinch, power, tripod, extension) and rest state and its performance is compared with a state-of-the-art Electromyography (EMG) bioamplifier. The decoding performance of the decoding models trained on the data acquired from the armband is significantly better than the performance of the models trained on raw EMG data. The hardware design and the related processing software, are disseminated in an open-source manner.

Over the last decade, adaptive tendon driven devices have gained an increased interest from the research community for their lightweight, compact, and affordable design features attributed to the utilisation of underactuation, differential mechanisms, and structural compliance. Although adaptive tendon driven devices are capable of efficiently executing stable grasps under significant object pose uncertainties with simplistic control algorithms, they lack the controllability over individual fingers in comparison to traditional fully actuated designs. In this paper, we focus on the development of a selectively lockable differential mechanism that is powered through a small and low torque servo to provide increased autonomy to highly underactuated and adaptive prosthetic hands, without compromising the weight, cost, and compactness of the device. The proposed prosthetic hand is experimentally validated through four tests: i) grasping posture and gesture execution experiments, ii) grasping experiments with everyday life objects, iii) force exertion experiments, and iv) Electromyography (EMG) based control of the prosthetic hand.

The ongoing global healthcare crisis has amplified the need for automation of manual tasks in several industries and service sectors. Simple household tasks such as tidying and cleaning are in high demand, with only a few robotic platforms capable of performing them due to the mobility, workspace, and dexterity requirements. This work presents ARoA, an autonomous robotic assistant that can execute complex tasks in industrial, service, and home environments. It is equipped with two lightweight, compliant, 7 degree of freedom arms and a pair of adaptive end-effectors that enable efficient execution of a wide range of tasks. Due to the linear rail based torso system that supports the arms, the ARoA offers exceptional flexibility in terms of reachable workspace. A framework for vision-based execution of tidying and cleaning tasks is also proposed and integrated in the platform. The efficiency of the ARoA platform was experimentally validated through two everyday life applications: i) picking up and tidying randomly scattered household objects and ii) cleaning of common surfaces.

Electromyography (EMG) based interfaces have been used in various robotics studies ranging from teleoperation and telemanipulation applications to the EMG based control of prosthetic, assistive, or robotic rehabilitation devices. But most of these studies have focused on the decoding of user's motion or on the control of the robotic devices in the execution of simple tasks (e.g., grasping tasks). In this work, we present a learning scheme that employs High Density Electromyography (HD-EMG) sensors to decode a set of dexterous, in-hand manipulation motions (in the object space) based on the myoelectric activations of human forearm and hand muscles. To do that, the subjects were asked to perform roll, pitch, and yaw motions manipulating two different cubes. The first cube was designed to have a center of mass coinciding with the geometric center of the cube, while for the second cube the center of mass was shifted 14 mm to the right (off-centered design). Regarding the acquisition of the myoelectric data, custom HD-EMG electrode arrays were designed and fabricated. Using these arrays, a total of 89 EMG signals were extracted. The object motion decoding was formulated as a regression problem using the Random Forests (RF) technique and the muscle importances were studied using the inherent feature variables importance calculation procedure of the RF. The muscle importance results show that different subjects use different strategies to execute the same motions on same object when the weight is off-centered. Finally, the decoded motions were used to control a five fingered robotic hand in a proof-of-concept application.

Over the last years, hand exoskeletons have become a popular and efficient technical solution for assisting people that suffer from neurological and musculoskeletal diseases and enhance the capabilities of healthy individuals. These devices can vary from rigid and complex structures to soft, lightweight, wearable gloves. Despite the significant progress in the field, most existing solutions do not provide the same dexterity as the healthy human hand. In this paper, we focus on the development of a hybrid (tendon-driven and pneumatic), lightweight, affordable, wearable exoskeleton glove equipped with abduction/adduction capabilities and a pneumatic telescopic extra thumb that increases grasp stability. The efficiency of the proposed device is experimentally validated through three different types of experiments: i) abduction/adduction tests, ii) force exertion experiments that capture the maximum forces that can be applied by the proposed device, and iii) grasp quality assessment experiments that focus on the effect of the inflatable thumb on enhancing grasp stability. The hybrid assistive glove considerably improves the grasping capabilities of the user, being able to exert the forces required to assist people in the execution of activities of daily living.

Current trends in industrial automation favor agile systems that allow adaptation to rapidly changing task requirements and facilitate customized production in smaller batches. This work presents a flexible manufacturing system relying on compliance control, CAD based localization, and a multi-modal gripper to enable fast and efficient task programming for assembly operations. CAD file processing is employed to extract component pose data from 3D assembly models, while the system's active compliance compensates for errors in calibration or positioning. To minimize retooling delays, a novel gripper design incorporating both a parallel jaw element and a rotating module is proposed. The developed system placed first in the manufacturing track of the Robotic Grasping and Manipulation Competition of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2019, experimentally validating its efficiency.

To facilitate the development of a new generation of Virtual Reality systems and their introduction in everyday life applications, new intuitive, immersive methods of interfacing have to be developed. Over the years, Electromyography (EMG) based interfaces have been utilized for unobtrusive interaction with computer systems. However, previous EMG studies have not explored the continuous decoding of the effects of human motion (e.g., manipulated object behavior) in simulated and virtual environments. In this work, we present an EMG based learning framework that can allow for an immersive interaction with Virtual Reality environments. To do that, EMG activations from the muscles of the forearm and the hand were acquired during the execution of object manipulation tasks in a virtual world along with the motion of the object. The virtual world was visualized using an HTC Vive VR headset, while the hand motions were tracked with a dataglove equipped with magnetic motion capture sensors. The object motion decoding was formulated as a regression problem using the Random Forests methodology. The study shows that the object motion can be successfully decoded using the EMG activations, despite the lack of haptic feedback.

With the emergence of agile manufacturing in highly automated industrial environments, the demand for efficient robot adaptation to dynamic task requirements is increasing. For assembly tasks in particular, classic robot programming methods tend to be rather time intensive. Thus, effectively responding to rapid production changes requires faster and more intuitive robot teaching approaches. This work focuses on combining programming by demonstration with path optimization and local replanning methods to allow for fast and intuitive programming of assembly tasks that requires minimal user expertise. Two demonstration approaches have been developed and integrated in the framework, one that relies on human to robot motion mapping (teleoperation based approach) and a kinesthetic teaching method. The two approaches have been compared with the classic, pendant based teaching. The framework optimizes the demonstrated robot trajectories with respect to the detected obstacle space and the provided task specifications and goals. The framework has also been designed to employ a local replanning scheme that adjusts the optimized robot path based on online feedback from the camera-based perception system, ensuring collision-free navigation and the execution of critical assembly motions. The efficiency of the methods has been validated through a series of experiments involving the execution of assembly tasks. Extensive comparisons of the different demonstration methods have been performed and the approaches have been evaluated in terms of teaching time, ease of use, and path length.

The concept of teleoperation has been studied since the advent of robotics and has found use in a wide range of applications, including exploration of remote or dangerous environments (e.g., space missions, disaster management), telepresence based time optimisation (e.g., remote surgery) and robot learning. While a significant amount of research has been invested into the field, intricate manipulation tasks still remain challenging from the user perspective due to control complexity. In this paper, we propose an intuitive, affordances oriented telemanipulation framework for a dual robot arm hand system. An object recognition module is utilised to extract scene information and provide grasping and manipulation assistance to the user, simplifying the control of adaptive, multi-fingered hands through a commercial Virtual Reality (VR) interface. The system's performance was experimentally validated in a remote operation setting, where the user successfully performed a set of bimanual manipulation tasks.

Teleoperation and telemanipulation have since the early years of robotics found use in a wide range of applications, including exploration, maintenance, and response in remote or hazardous environments, healthcare, and education settings. As the capabilities of robot manipulators grow, so does the control complexity and the remote execution of intricate manipulation tasks still remains challenging for the user. This paper proposes an intuitive telemanipulation framework based on electromyography (EMG) and fiducial marker based tracking that can be used with a dexterous robot arm hand system. The EMG subsystem captures the myoelectric activations of the user during the execution of specific hand postures and gestures and translates them into the desired grasp type for the robot hand. The pose of the tracked fiducial marker is used as a task-space goal for the robot end-effector. The system performance is experimentally validated in a remote operation setting, where the system successfully performs a telemanipulation task.

Manual palpation is commonly used to localize tumors and other features buried deep inside organs during open surgery. This approach is not feasible in minimally invasive or robotic surgery, as the contact with the tissue is mediated by instruments. To address this problem, we propose a soft robotic skin (SRS) that can be deployed from a small incision and create a stiffness map in a single step. Such a skin is composed of a matrix of soft robotic tactile elements (SRTEs), each one able to expand and record the tissue response during expansion. In this paper, we firstly prove the feasibility of palpation using a single SRTE. Then, we present and test a soft-suction based anchoring mechanism able to keep the SRS in the desired position in contact with the tissue, allowing surgeons to palpate different sides of the organ. Finally, we detail a calibration method for the SRTE, and assess the feasibility of identifying lumps buried inside a soft tissue phantom, and then inside a chicken liver during an ex-vivo trial. Experimental results show that the SRTE was able to differentiate simulated lumps (up to 3.25 mm deep) from healthy tissue in both the phantom and the ex-vivo trials. These results, added to the ability of the suction gripper to compensate for the expansion forces of each SRTE, are paving the way for soft robotic autonomous tools that can be used for intraoperative mapping of tissue cancers.