Manufacturing is shifting from autonomous robots to collaborative robots (cobots) that work alongside humans. To foster effective human-robot collaboration, the robot must be aware of the operator’s needs and be able to adapt accordingly. A fundamental step in this collaboration is the robot’s ability to recognize and understand the operator’s actions online. This study aims to develop a computer vision system to recognize and classify human actions in a collaborative assembly scenario. The goal has been to create a lightweight action recognition model, by investigating various models, including Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks, combined with YOLO-based object recognition. This work presents several model architectures designed to classify human actions based on different types of input data. Specifically, the models process hand landmark data from an RGB-D camera. Four actions commonly found in manufacturing environments are classified. A custom dataset was created from scratch to support this research. From the analysis of the different approaches, the best-performing model achieved an offline action recognition accuracy of 95%, and an online accuracy exceeding 90%. Comparing different methodologies and input data for human action recognition helps to highlight their strengths and limitations, enabling the identification of the most effective strategy for recognizing actions in human-robot collaboration. This approach contributes to significant advancements in developing more intuitive and efficient collaborative environments within manufacturing settings. Data, code, and results are available at: https://github.com/grgzpp/human-action-recognition-and-anticipation.

Physical human-robot interaction (pHRI) remains a key challenge for achieving intuitive and safe interaction with robots. Current advancements often rely on external tactile sensors as interface, which increase the complexity of robotic systems. In this study, we leverage the intrinsic tactile sensing capabilities of collaborative robots to recognize digits drawn by humans on an uninstrumented touchpad mounted to the robot’s flange. We propose a dataset of robot joint torque signals along with corresponding end-effector (EEF) forces and moments, captured from the robot’s integrated torque sensors in each joint, as users draw handwritten digits (0–9) on the touchpad. The pHRI-DIGI-TACT dataset was collected from different users to capture natural variations in handwriting. To enhance classification robustness, we developed a data augmentation technique to account for reversed and rotated digits inputs. A Bidirectional Long Short-Term Memory (Bi-LSTM) network, leveraging the spatiotemporal nature of the data, performs online digit classification with an overall accuracy of 94% across various test scenarios, including those involving users who did not participate in training the system. This methodology is implemented on a real robot in a fruit delivery task, demonstrating its potential to assist individuals in everyday life. Dataset and video demonstrations are available at: https://TS-Robotics.github.io/pHRI-DIGI/.

Soft robots rely on soft actuators, whose nonlinear responses are challenging to model, simulate, and integrate into designs. This complexity hinders the development of advanced soft robots and rigid mechanisms that require soft actuators for compliant actuation. To address this, in this study we present a framework for model development and actuator integration that offers both real and virtual environments for testing and validation. This framework includes high-fidelity digital twins and an actuator integration bench. The digital twin allows for the testing of virtual actuator models in a validated digital environment, replicating various load profiles. The actuator integration bench provides a safe, reproducible platform for validating both models and controllers under different loads, load profiles, and inertial conditions. Together, these tools enable rapid and reliable validation of actuator models and controllers, accelerating the development cycle of complex soft robots. We conclude by demonstrating the proposed workflow using liquid-gas phase transition actuators as a demonstrative test subject. Our source code is available at: https://github.com/softrobotic/antagonistic.

The key challenge in expanding human-robot collaboration is enabling robots to understand and adapt to human needs, fostering seamless interaction. This research develops an open-source framework integrated with a collaborative robotic arm to recognize, predict, and anticipate human actions in an assembly scenario. Human action classification was performed by comparing various approaches, including Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks, combined with YOLO-based object recognition. The study utilized a pneumatic cylinder as an example component for the assembly, along with the KUKA LBR iiwa 7 R800 robotic arm. The action recognition model, developed from scratch with a custom dataset, achieved 95% offline accuracy and over 90% online. The action prediction model, trained on real assembly sequences from human demonstrations, suggests the next robot action. This approach improves the flexibility and customization by allowing the assembly sequence to be learned directly observing the operator performing the task, without prior knowledge (except for the objects to detect). This advancement enhances cobots ability to recognize, predict, and anticipate human actions, improving intuitive and efficient collaboration in manufacturing.

Force and proximity sensors are key in robotics, especially when applied in collaborative robots that interact physically or cognitively with humans in real unstructured environments. However, most of the existing sensors for use in robotics are limited by: 1) their scope, measuring single parameters/events and often requiring multiple types of sensors; 2) being expensive to manufacture, limiting their use to where they are strictly necessary and often compromising redundancy; and 3) have null or reduced physical flexibility, requiring further costs with adaptation to a variety of robot structures. This article presents a novel mechanically flexible force and proximity hybrid sensor based on piezoresistive and self-capacitive phenomena. The sensor is inexpensive and easy to apply even on complex-shaped robot structures. The manufacturing process is described, including controlling circuits, mechanical design, and data acquisition. Experimental trials featuring the characterization of the sensor were conducted, focusing on both force–electrical resistance and self-capacitive proximity response. The sensor’s versatility, flexibility, thinness (1-mm thickness), accuracy (reduced drift), and repeatability demonstrated its applicability in several domains. Finally, the sensor was successfully applied in two distinct situations: hand-guiding a robot (by touch commands) and human–robot collision avoidance (by proximity detection).

Redundancy in robotic manipulators has many advantages. It is successfully used to achieve better dexterity, and to avoid obstacles, singularities, or the kinematic limitations. However, redundancy makes the inverse kinematics (IK) problem harder to solve. The damped least squares (DLS) is a powerful method for calculating the IK of redundant robots, but it suffers from noncyclicity issue, where a closed curve motion in the Cartesian space of the end-effector (EEF) does not map into a closed curve in the joint space. This results in nonrepetitive motion in the joint space, even though the EEF motion is repetitive. In this article, we present a solution for the noncyclicity problem in the DLS method. The proposed scheme was successfully tested both in simulation (9 DoF robot) and on a real robotic manipulator (7 DoF robot).

Flexibility, adaptability and standardization of multidisciplinary production processes are key issues for today's industry. The digitalization of industry partially helps to overcome these challenges, leading to the need for efficient management of data. AutomationML has been pointed as a solution to solve the problem of data exchange between heterogeneous engineering tools landscape. This paper introduces a practical approach for data exchange on a production-engineering environment linked to Metal Additive Manufacturing (MAM). The proposed approach allows the exchange of data/information between different engineering tools using AutomationML Engine. For example, the MAM paths can be edited and enriched with information along the different stages of the process (design, simulation, robotics) using a neutral format. In the sphere of Direct Energy Deposition (DED) technologies it is proposed a practical use case for data exchange and editing, from computer aided design (CAD), to path planning, to process parameters definition, to robot programming. Results demonstrated the effectiveness of the proposed AutomationML-based solution.

The use of gestures as interface between humans and robots to facilitate communication between them is a long-sought goal. Although many gesture solutions have been presented, none of them cope entirely with wrong gesture recognition. This study proposes a novel electromyography (EMG) prototype sensor to capture gestures and also algorithms and procedures to discriminate data containing valid gestures (segmentation). Gestures are recognized using convolutional neural network (CNN) model. The proposed solution presented high recognition accuracy overcoming other similar studies in literature. Test results demonstrated that the proposed solution presents high performance and suggested its use in industrial environment.

Rand-guiding of collaborative redundant manipulators allows an unskilled user to interact and program the robot intuitively. Many industrial applications require precise positioning at the end-effector (EEF) level inside cluttered environments, where manipulator's redundancy is required. Yet, the potentialities of redundancy while hand-guiding at EEF level are not fully explored. This paper addresses the subject of precision in hand-guiding at EEF level while using the redundancy for in-contact obstacle navigation. In the presence of a contact with an obstacle, the proposed null space control method actuates in a way that the manipulator slides compliantly with its structure on the body of the obstacle while preserving the precision of the hand-guiding motion at EEF level. Force/torque (FT) data from a FT sensor mounted at the robot flange are the input for EEF precision hand-guiding while the torque data from the joints of the manipulator represent the contact between robot structure and obstacles. Experimental tests were carried out successfully using a KUKA iiwa industrial manipulator with 7 degrees of freedom (DOF). Where, the EEF is hand-guided on a straight line while the robot is sliding on the obstacle with its structure, results indicate the precision of the proposed method.

In this paper we present a novel strategy and implementation of autonomous navigation for the Pepper robot. The proposed solution is modular and relies on the proper integration of existing Pepper functionalities. The human interacts with the robot using voice and/or gesture commands to setup the robot functionalities. An example use case demonstrates the ability of the robot to navigate in office environment and grasp an object to place in a target location. The robot is also able to avoid obstacles while navigating.

Object recognition has attracted increasing attention of researchers due to its numerous applications. For instance, it enables robots to carry out tasks like searching for an object in an unstructured environment or retrieving a tool for a human co-worker. In this study, we present a new technique for unsupervised feature extraction from red, green, blue, plus depth (RGB-D) data, which is then combined with several classifiers to perform object recognition. Specifically, our architecture segments all objects in a table top scene through an unsupervised clustering technique. It focuses separately on each object to extract both shape and visual features. We conduct experiments on a subset of 20 objects selected from the YCB object and model set and evaluate the performance of several classifiers. The most effective one achieves an accuracy of 99.7% when trained and tested on samples acquired with the same conditions (equipment and environment). Results degrade when the system is trained with YCB data and tested with data acquired from a Kinect sensor in online laboratorial implementation.

A flexible programming and orchestration system for human-robot collaborative tasks is proposed. Five different interaction modes are suggested to test two Task-Managers (TMs) acting as orchestrators between a human co-worker and a robot. Both TMs rely on the task-based programming concept providing modular and scalable capabilities, allowing robot code reuse, fast robot programming and high robot programming flexibility. The TMs provide visual and audio feedback to the user about the robot task sequence being executed, guiding the user during the iterative process. The interaction modes tested were: (1) human arm gestures, (2) human hand gestures, (3) physical contact between human and robot, and (4–5) two hybrid interaction modes combining each one of the two first interaction modes with the last one. Experimental tests indicated that users prefer fast interactions with small number of interaction items to higher flexibility. Both TMs provide intuitive and modular interface for collaborative robots with a human in the loop.

Increasingly, robots have more degrees of freedom (DOF), imposing a need for calculating more complex dynamics. As a result, better efficiency in carrying out dynamics computations is becoming more important. In this study, an efficient method for computing the joint space inertia matrix (JSIM) for high DOF serially linked robots is addressed. We call this method the Geometric Dynamics Algorithm for High number of robot Joints (GDAHJ). GDAHJ is non-symbolic, preserve simple formulation, and it is convenient for numerical implementation. This is achieved by simplifying the way to recursively derive the mass-matrix exploiting the unique property of each column of the JSIM and minimizing the number of operations with O(n2) complexity. Results compare favorably with existing methods, achieving better performance over state-of-the-art by Featherstone when applied for robots with more than 13 DOF.

Continuous and real-time gesture spotting is a key factor in the development of novel human-machine interaction modalities. Gesture recognition can be greatly improved with previous reliable segmentation. This paper introduces a new unsupervised threshold-based hand/arm gesture segmentation method to accurately divide continuous data streams into dynamic and static segments from unsegmented and unbounded input data. This segmentation may reduce the number of wrongly classified gestures in real-world conditions. The proposed approach identifies sudden inversions of movement direction, which are a cause of oversegmentation (excessive segmentation). This is achieved by the analysis of velocities and accelerations numerically derived from positional data. A genetic algorithm is used to compute feasible thresholds from calibration data. Experimental tests with three different subjects demonstrated an average oversegmentation error of 2.70% in a benchmark for motion segmentation with a feasible sliding window size.

Continuous and real-time gesture spotting is a key factor in the development of novel human-robot interaction (HRI) modalities. Gesture recognition can be greatly improved with previous reliable segmentation. This paper introduces a new unsupervised threshold-based gesture segmentation method to accurately divide continuous data streams into dynamic and static blocks, without previous knowledge of gesture data (unbounded input data). This type of segmentation may reduce the number of wrongly classified gestures. The proposed approach identifies sudden inversions of movement direction which are a cause of over segmentation. This is achieved by the analysis of velocities and accelerations numerically derived from positional data. A genetic optimization algorithm is used to optimize thresholds from calibration data. Experimental tests were based on the application of the proposed method using a data glove and a magnetic tracking device as interaction technology. The over-segmentation error was 5.5% in a benchmark for motion segmentation composed of samples retrieved from two subjects.

Continuous and real-time gesture spotting is a key factor for the development of novel Human-Robot Interaction (HRI) modalities and further push the use of robots in our society. In this paper we present a hand gesture recognition module for large vocabularies of static and dynamic gestures, with limited training. The recognition module uses feature-samples obtained with an automatic motion detection-based segmentation algorithm, being the source data obtained from a magnetic tracker for the wrist and a data glove for the hand. The classifiers proposed are Multi-Layer Neural Networks (Perceptrons) (MLP) with one or two hidden-layers, with an accuracy of 98.7% for 25 Static Gestures (SGs) and up to 99.0% for 10 Dynamic Gestures (DGs). The results are on par or better than similar studies.

This article presents an inchworm climbing robot that is designed with switchable magnets and a single DOF arm. The InchwormClimber works on ferromagnetic structures and consumes little energy when climbing and can descend safely with almost zero energy consumption. Furthermore, considering the critical role of the adhesion unit in the overall functionality of the robot (weight, climbing speed and the payload), we optimized the switchable magnet unit for a higher adhesion force per mass unit.

The goal of this work is to study the application of switchable magnets (SM) for climbing robots. A switchable magnet is a device which uses moving permanent magnets to change the magnetic flux path and switch on or off the magnetic attraction force. In our work we used Comsol Multiphysics, a physics simulation software in order to simulate the flow of the magnetic flux on switchable magnets on its different states and study the effect of different design and material parameters on the attraction force of the unit. Bearing in mind the lessons learned from this study, we developed a novel device in a smaller scale with the best holding force/mass ratio, for using in climbing robot applications. As a case study, three of these optimized SM units are then equipped with an actuator that can rotate the moving magnets, turning the device on and off. This new device is employed in a novel adaptive adhesion unit for the OmniClimber robot, replacing its previous system which was relying on electromagnets magnets. We demonstrate this novel device application on a climbing robot and the advantages relative to electromagnet or permanent magnet based devices.

The relevance and importance of industrial robots in manufacturing has increased over the years, with applications in diverse new and non-traditional manufacturing processes. This paper presents the concept and design of a novel friction stir welding (FSW) robotic platform for welding polymeric materials. It was conceived to have a number of advantages over common FSW machines: it is more flexible, cheaper, easy and fast to setup, and easy to program. The platform is composed by three major groups of hardware: a robotic manipulator, a FSW tool and a system that links the manipulator wrist to the FSW tool (support of the FSW tool). This system is also responsible for supporting a force/torque (F/T) sensor and a servo motor that transmits motion to the tool. During the process, a hybrid force/motion control system adjusts the robot trajectories to keep a given contact force between the tool and the welding surface. The platform is tested and optimized in the process of welding acrylonitrile butadiene styrene (ABS) plates. Experimental tests proved the versatility and validity of the solution.

New and more natural human-robot interfaces are of crucial interest to the evolution of robotics. This paper addresses continuous and real-time hand gesture spotting, i.e., gesture segmentation plus gesture recognition. Gesture patterns are recognized by using artificial neural networks (ANNs) specifically adapted to the process of controlling an industrial robot. Since in continuous gesture recognition the communicative gestures appear intermittently with the non-communicative, we are proposing a new architecture with two ANNs in series to recognize both kinds of gesture. A data glove is used as interface technology. Experimental results demonstrated that the proposed solution presents high recognition rates (over 99% for a library of ten gestures and over 96% for a library of thirty gestures), low training and learning time and a good capacity to generalize from particular situations.

This paper introduces a new approach to 3-D position estimation from acceleration data, i.e., a 3-D motion tracking system having a small size and low-cost magnetic and inertial measurement unit (MIMU) composed by both a digital compass and a gyroscope as interaction technology. A major challenge is to minimize the error caused by the process of double integration of accelerations due to motion (these ones have to be separated from the accelerations due to gravity). Owing to drift error, position estimation cannot be performed with adequate accuracy for periods longer than few seconds. For this reason, we propose a method to detect motion stops and only integrate accelerations in moments of effective hand motion during the demonstration process. The proposed system is validated and evaluated with experiments reporting a common daily life pick-and-place task.

Increasingly, industrial robots are being used in production systems. This is because they are highly flexible machines and economically competitive with human labor. The problem is that they are difficult to program. Thus, manufacturing system designers are looking for more intuitive ways to program robots, especially using the CAD drawings of the production system they developed. This paper presents an industrial application of a novel CAD-based off-line robot programming (OLP) and simulation system in which the CAD package used for cell design is also used for OLP and robot simulation. Thus, OLP becomes more accessible to anyone with basic knowledge of CAD and robotics. The system was tested in a robot-assisted sheet metal bending cell. Experiments allowed identifying the pros and cons of the proposed solution.

Most of industrial robots are still programmed using the typical teaching process, through the use of the robot teach pendant. In this paper is proposed an accelerometer-based system to control an industrial robot using two low-cost and small 3-axis wireless accelerometers. These accelerometers are attached to the human arms, capturing its behavior (gestures and postures). An Artificial Neural Network (ANN) trained with a back-propagation algorithm was used to recognize arm gestures and postures, which then will be used as input in the control of the robot. The aim is that the robot starts the movement almost at the same time as the user starts to perform a gesture or posture (low response time). The results show that the system allows the control of an industrial robot in an intuitive way. However, the achieved recognition rate of gestures and postures (92%) should be improved in future, keeping the compromise with the system response time (160 milliseconds). Finally, the results of some tests performed with an industrial robot are presented and discussed.