Human Action Recognition (HAR) plays a vital role in manufacturing assembly tasks, addressing key areas such as worker safety, operational support, production optimization, employee training, and facilitating human-robot collaboration. This paper introduces a skeleton-based action recognition approach based on a CNN deep neural network architecture. Joint-to-joint distances are used to represent human movements during assembly tasks, enabling the model to capture intricate motion patterns. The primary focus of this work is on structuring the input data in various ways to analyze how these variations influence the network performance. Studying the spatial configurations of input data for human action recognition in an assembly task is an insightful and challenging research topic. In assembly tasks, the high similarity between actions and the operator-specific execution variations make distinguishing actions more complex. This work investigates how the arrangement of input data impacts model accuracy. In particular, two input data configurations are analyzed: onechannel and multi-channel types. The assembly actions are classified using a CNN-based architecture. So, the different data configurations directly influence the type of CNN applied, which can be 2D or 3D. The proposed approach is evaluated on the publicly available HA4M dataset. The obtained results showed that the proposed data structure greatly influences the model performance measurement.

Human Action Recognition (HAR) in industrial assembly scenarios presents significant challenges, primarily due to the slight differences in motion patterns across fine-grained actions. In this work, we address the problem of action recognition in assembly tasks by employing skeleton data to represent detailed human movements. To effectively capture the spatial and temporal dependencies among joints, we apply a Transformer-based architecture. We conducted an extensive evaluation by varying the model dimension to analyze its effect on recognition performance. Furthermore, given the high semantic and structural similarity between certain action classes, we propose a class merging strategy that combines highly similar actions into unified categories. This not only simplifies the classification task, but also improves overall recognition performance by reducing ambiguity. Experimental results demonstrate the effectiveness of Transformers for fine-grained action recognition in industrial settings, highlighting the importance of both architectural tuning and label refinement when dealing with closely related human actions.

With Industry 5.0, operators’ physical and cognitive behavior is crucial in any field, particularly manufacturing and production lines. To this end, the need to monitor humans performing specific tasks when working alongside robotic systems has grown further. Guaranteeing the well-being of operators sharing a workspace with industrial robots can drastically reduce risky and harmful situations for the operators while leading the robot to adapt to humans fully. For this purpose, monitoring systems can be very helpful in studying the most suitable deep learning methodologies to obtain information from the movements of the individual operator performing a specific task. Therefore, action segmentation can be fundamental to establishing a new and safe way of communication among operators and robots. In this work, a system for segmenting actions performed by operators assembling an industrial object is developed. The public HA4M dataset has been used to train and test temporal action segmentation models by using MS-TCN++ architecture. Highly discriminant features have been extracted from the dataset, and different training approaches based on multimodal data, including RGB and skeletal joints in Depth and RGB resolutions, have been considered. Results show the effectiveness of the proposed system, laying the foundation for further studies for detecting the operators’ actions in the challenging context of Human-Robot Interaction and Collaboration.

This paper introduces a dataset on Human-cobot collaboration for Action Recognition in Manufacturing Assembly (HARMA3). It is a collection of RGB frames, Depth maps, RGB-to-depth-Aligned (RGB-A) frames and Skeleton data relative to actions performed by different subjects in collaboration with a cobot for building an Epicyclic Gear Train (EGT). In particular, 27 subjects executed several trials of the assembly task, which consisted of 7 actions. Data were collected in a laboratory scenario using two Microsoft®Azure Kinect cameras positioned in frontal and lateral positions. The dataset represents a good foundation for developing and testing advanced action recognition as well as action segmentation systems with far-reaching implications beyond human-cobot collaboration. Further potential applications include Computer Vision, Machine Learning, and Smart Manufacturing. Preliminary experiments for action segmentation by applying a state-of-the-art method on features extracted from RGB and skeletal data are presented in this paper, showing high-performance rates.

This paper reviews the recent literature on technologies and methodologies for quantitative human gait analysis in the context of neurodegenerative diseases. The use of technological instruments can be of great support in both clinical diagnosis and severity assessment of these pathologies. In this paper, sensors, features and processing methodologies have been reviewed in order to provide a highly consistent work that explores the issues related to gait analysis. First, the phases of the human gait cycle are briefly explained, along with some non-normal gait patterns (gait abnormalities) typical of some neurodegenerative diseases. Then the paper reports the most common processing techniques for both feature selection and extraction and for classification and clustering. Finally, a conclusive discussion on current open problems and future directions is outlined.

Service robots are expected to be used in many household in the near future, provided that proper interfaces are developed for the human robot interaction. Gesture recognition has been recognized as a natural way for the communication especially for elder or impaired people. With the developments of new technologies and the large availability of inexpensive depth sensors, real time gesture recognition has been faced by using depth information and avoiding the limitations due to complex background and lighting situations. In this paper the Kinect Depth Camera, and the OpenNI framework have been used to obtain real time tracking of human skeleton. Then, robust and significant features have been selected to get rid of unrelated features and decrease the computational costs. These features are fed to a set of Neural Network Classifiers that recognize ten different gestures. Several experiments demonstrate that the proposed method works effectively. Real time tests prove the robustness of the method for realization of human robot interfaces.

Distributed networks of sensors have been recognized to be a powerful tool for developing fully automated systems that monitor environments and human activities. Nevertheless, problems such as active control of heterogeneous sensors for high-level scene interpretation and mission execution are open. This paper presents the authors' ongoing research about design and implementation of a distributed heterogeneous sensor network that includes static cameras and multi-sensor mobile robots. The system is intended to provide robot-assisted monitoring and surveillance of large environments. The proposed solution exploits a distributed control architecture to enable the network to autonomously accomplish general-purpose and complex monitoring tasks. The nodes can both act with some degree of autonomy and cooperate with each other. The paper describes the concepts underlying the designed system architecture and presents the results obtained working on its components, including some simulations performed in a realistic scenario to validate the distributed target tracking algorithm.

People tracking is a central and crucial point for the development of intelligent surveillance systems. When multiple cameras are used, the problem becomes more challenging as people re-identification is needed. Humans can greatly change their appearance according to posture, clothing and lighting conditions, thus defining features that describe people moving in large scenarios is a complex task. In this paper the problem of people re-identification and tracking is reviewed. The most used methodologies are discussed and insight into open problems and future research directions is provided.

Passive Radio Frequency Identification (RFID) is being increasingly used in mobile robotics applications, as it provides inexpensive and effective solutions to data association issues in basic navigation tasks. Nonetheless, problems related to sensitivity of the signal to interference and reflections, and missing tag range and bearing information are open. In this paper, we propose a novel approach to passive RFID, which tackles those issues using fuzzy reasoning. Specifically, first, we present a fuzzy antenna model. Then, based on this model, we describe two fuzzy logic methods for tag localization. One allows us to accurately localize passive tags in the environment and to generate what we call an RFID-augmented map; the other is suited for estimating the bearing of a tag relative to the robot. The general use of both methods is in object localization, map building, environment monitoring, and robot pose estimation. Results of experimental tests demonstrate that fuzzy logic is appropriate to operate under uncertainty in RFID systems, and allows for accurate tag localization.

In this paper, we present a mobile robot global positioning system which combines passive RFID and monocular vision. The method is intended to solve the so-called kidnapped robot problem, that is, the problem of estimating the robot pose when it is first booted up or it has been kidnapped and taken to some unknown location. The main novelty of the approach is the use of fuzzy logic to cope with both uncertainty in RFID data and missing tag position information. Fuzzy logic allows us to manage inaccurate sensor data using rules easily understandable by humans. An additional advantage of the proposed method is that it does not require the robot to move around to solve ambiguities, nor does it need several tags to be properly arranged. Conversely, only one passive RFID tag and a single visual landmark are sufficient to univocally estimate the pose of the robot in the environment.

In this work the complex behavior of localizing a mobile vehicle with respect to the door of the environment and then reaching the door has been developed. The robot uses visual information to detect and recognize the door and to determine its state with respect to it. This complex task has been divided into two separate behaviors: door-recognition and door-reaching. A supervised methodology based on learning by components has been applied for recognizing the door. Learning by components allows to recognize the door also in difficult situations such as partial occlusions and besides, it makes recognition independent of viewpoint variations and scale changes. An unsupervised methodology based on reinforcement learning has been used for the door-reaching behavior, instead. The image of the door gives information about the relative position of the vehicle with respect to the door. Then the Q-learning algorithm is used to generate the optimal state-action associations. The problem of defining the state and the action sets has been addressed with the aim of producing smooth paths, of reducing the effects of visual errors during real navigation, and of keeping low the computational cost during the learning phase. A novel way to obtain a continuous action set has been introduced: it uses a fuzzy model to evaluate the system state. Experimental results in real environment show both the robustness of the door-recognition behavior and the generality of the door-reaching behavior.

This paper presents an omnidirectional distributed vision system that learns to navigate a robot in an office-like environment without any knowledge about the calibration of the cameras or the robot control law. The system is composed of several omnidirectional vision agents (implemented with an omnidirectional camera and a computer). The first vision agent learns to control the robot with SARSA(/spl lambda/) reinforcement learning, using the LEM strategy to speed-up learning. Once the first vision agent learnt the correct policy, it transfers its knowledge to the other vision agents. The other vision agents might have different intrinsic and extrinsic camera parameters (that are unknown), so a certain amount of re-learning is needed. Reinforcement learning is well suited for this. In this paper, we present the structure of the learning system and the discussion about the optimal values for the learning parameters. During the experimentation the learning phase of the first agent has been carried out, then the knowledge propagation and the re-learning stage of three different agents have been tested. The experimental results demonstrate the feasibility of the approach and the possibility to port the system on the actual robot and cameras.

The Q-learning algorithm, for its simplicity and well-developed theory, has been largely used in the last years in order to realize different behaviors for autonomous vehicles. The most frequent applications required the standard tabular formulation with discrete sets of state and action. In order to consider continues variables, function approximators such as neural networks are required. In this work we investigate the neural approach of Q-learning on the robot navigation task of wall following. Some issues have been addressed in order to deal with the convergence problem and the need of huge training sets. The experience replay paradigm has been also applied to reduce the unlearning problem. Two different neural network architectures, which use different spatial decompositions of the sensory input, have been compared. The aim is to investigate how different choices of architecture can affect the learning convergence, the optimality of the final controller and the generalization ability.

This paper presents a vision-based technique for detecting targets of the environment which have to be reached by an autonomous mobile robot during its navigational tasks. The targets the robot has to reach are the doors of the authors' office building. The detection of the door has been performed by detecting its most significant components in the image and it is based on data classification. Two neural classifiers have been trained for recognizing single components of the door. Then a combining algorithm, based on heuristic considerations, checks that they are in the proper geometric configuration of the structure of the door. The novelty of this work is to use together colour and shape information for identifying features and for detecting the components of the target. The approach, based on learning by components, is able to cleverly solve the problems of scale changes, perspective variations and partial occlusions. The obtained detecting system has been tested on a large test set of real images showing a high reliability and robustness: doors of different rooms, under different illumination conditions and by different viewpoints have been successfully recognized. Results in terms of door detection rate and false positive rate are presented throughout the paper.

A problem related to the use of reinforcement learning (RL) algorithms on real robot applications is the difficulty of measuring the learning level reached after some experience. Among the different RL algorithms, the Q-learning is the most widely used in accomplishing robotic tasks. The aim of this work is to a priori evaluate the optimal Q-values for problems where it is possible to compute the distance between the current state and the goal state of the system. Starting from the Q-learning updating formula the equations for the maximum Q-weights, for optimal and non-optimal actions, have been computed considering delayed and immediate rewards. Deterministic and non deterministic grid-world environments have been also considered to test in simulations the obtained equations. Besides the convergence rates of the Q-learning algorithm have been compared using different learning rate parameters.

We present a learning algorithm that realizes a simple goal-reaching task for an autonomous vehicle when only visual information about the goal is provided. The robot has to reach a door from every position of the environment. The state of the system is based on visual information received by a TV camera placed on the mobile robot. The vision algorithm is able to determine the relative position between the vehicle and the door according to the slopes of the contour lines of the door. A learning phase is carried out in simulation to obtain the optimal state-action rules. The learned knowledge is then transferred on the real robot for the testing phase. Experimental results show the generality of the knowledge learned as the real robot is always able to execute its paths towards the door in different environments.

We describe a navigation approach, based on a reinforcement learning algorithm, that allows a mobile robot to move in an unknown indoor environment learning autonomously in a few trials the actions for reaching a particular goal location. The control architecture merges visual information and sonar readings to evaluate the state of the system to which the learning algorithm relates the best movement for reaching the goal. As a result, after limited experience, the robot learns efficient behavioral sequences and improves its learning incrementally. In addition the learning system adapts well to new situations of the environment or new task requirements. The results obtained in simulation and in real experiments, carried out in our laboratory, have shown that our system is tolerant to noise in sensor measurements and during each trial is always able to reach the goal.

Sparse optical flow estimated by an image sequence, often, is enough to obtain useful information about camera motion. Normally, matching among features on image sequence is estimated by radiometric similarity. When the camera is moving on a flat surface (planer navigation) additional constraints can be introduced in order to obtain a more robust estimation of the optical flow. In this paper, an energy based optimization method to compute a robust sparse optical flow using global constraints induced by camera motion, is proposed. Convergence is reached in few iterations, making the proposed method suitable for real-time applications.

Goal-oriented navigation of a mobile robot by landmark based techniques is a straightforward and suitable approach. Most of the methods would normally give an estimation of the position and orientation of the vehicle, but often they are not able to provide a good estimate of the uncertainty in the measurement. That information is useful in application where multisensor fusion is requested. Our method permits to determine the vehicle location and relative uncertainty, when its orientation is obtained by a heading sensor, using a visual landmark based method. The approach is straightforward and suitable for real time performance on general purpose hardware. Experimental results are provided by implementation on our autonomous mobile vehicle SAURO.