Human Action Recognition (HAR) plays a vital role in manufacturing assembly tasks, addressing key areas such as worker safety, operational support, production optimization, employee training, and facilitating human-robot collaboration. This paper introduces a skeleton-based action recognition approach based on a CNN deep neural network architecture. Joint-to-joint distances are used to represent human movements during assembly tasks, enabling the model to capture intricate motion patterns. The primary focus of this work is on structuring the input data in various ways to analyze how these variations influence the network performance. Studying the spatial configurations of input data for human action recognition in an assembly task is an insightful and challenging research topic. In assembly tasks, the high similarity between actions and the operator-specific execution variations make distinguishing actions more complex. This work investigates how the arrangement of input data impacts model accuracy. In particular, two input data configurations are analyzed: onechannel and multi-channel types. The assembly actions are classified using a CNN-based architecture. So, the different data configurations directly influence the type of CNN applied, which can be 2D or 3D. The proposed approach is evaluated on the publicly available HA4M dataset. The obtained results showed that the proposed data structure greatly influences the model performance measurement.

With Industry 5.0, operators’ physical and cognitive behavior is crucial in any field, particularly manufacturing and production lines. To this end, the need to monitor humans performing specific tasks when working alongside robotic systems has grown further. Guaranteeing the well-being of operators sharing a workspace with industrial robots can drastically reduce risky and harmful situations for the operators while leading the robot to adapt to humans fully. For this purpose, monitoring systems can be very helpful in studying the most suitable deep learning methodologies to obtain information from the movements of the individual operator performing a specific task. Therefore, action segmentation can be fundamental to establishing a new and safe way of communication among operators and robots. In this work, a system for segmenting actions performed by operators assembling an industrial object is developed. The public HA4M dataset has been used to train and test temporal action segmentation models by using MS-TCN++ architecture. Highly discriminant features have been extracted from the dataset, and different training approaches based on multimodal data, including RGB and skeletal joints in Depth and RGB resolutions, have been considered. Results show the effectiveness of the proposed system, laying the foundation for further studies for detecting the operators’ actions in the challenging context of Human-Robot Interaction and Collaboration.

This paper introduces a dataset on Human-cobot collaboration for Action Recognition in Manufacturing Assembly (HARMA3). It is a collection of RGB frames, Depth maps, RGB-to-depth-Aligned (RGB-A) frames and Skeleton data relative to actions performed by different subjects in collaboration with a cobot for building an Epicyclic Gear Train (EGT). In particular, 27 subjects executed several trials of the assembly task, which consisted of 7 actions. Data were collected in a laboratory scenario using two Microsoft®Azure Kinect cameras positioned in frontal and lateral positions. The dataset represents a good foundation for developing and testing advanced action recognition as well as action segmentation systems with far-reaching implications beyond human-cobot collaboration. Further potential applications include Computer Vision, Machine Learning, and Smart Manufacturing. Preliminary experiments for action segmentation by applying a state-of-the-art method on features extracted from RGB and skeletal data are presented in this paper, showing high-performance rates.

N owadays, non-destructive techniques (NDT) playa fundamental role in the production industry since early defects detection (EDD) can reduce possible costs and avoid catastrophic failures. Under these aspects, all methods for fast and reliable inspection deserve special attention. This paper proposes a method to detect manufacturing defects or other damage mechanisms without compromising the original condition of the material using active IR thermography and automatic semantic segmentation. The segmentation of defects in composite materials is achieved by using a deep learning algorithm on a high-variance dataset obtained performing lock-in thermography under five different heat source configurations. Experimental results on specimens with known defects have demonstrated that the proposed methodology provides satisfying performances in automatic defect detection.

This paper reviews the recent literature on technologies and methodologies for quantitative human gait analysis in the context of neurodegenerative diseases. The use of technological instruments can be of great support in both clinical diagnosis and severity assessment of these pathologies. In this paper, sensors, features and processing methodologies have been reviewed in order to provide a highly consistent work that explores the issues related to gait analysis. First, the phases of the human gait cycle are briefly explained, along with some non-normal gait patterns (gait abnormalities) typical of some neurodegenerative diseases. Then the paper reports the most common processing techniques for both feature selection and extraction and for classification and clustering. Finally, a conclusive discussion on current open problems and future directions is outlined.

This paper introduces the control architecture of a platform aimed at promoting good mental health for workers interacting with collaborative robots (cobots). The platform aim is to render industrial production cells capable of automatically adapting their behavior in order to improve the operator’s quality of experience and level of engagement and to minimize his/her psychological strain. In order to achieve such a goal, an extremely rich and complex framework is required. Starting from the identification of the parameters that could influence the collaboration experience, the envisioned human- driven control structure is presented together with a detailed description of the components required to implement such an automated system. Future works will include proper tuning of control parameters with dedicated experimental sessions, together with the definition of organizational and technical guidelines for the design of a mental-health-friendly cobot-based manufacturing workplace.

In this paper, a method to find, exploit and classify ambiguities in the results of a person re-identification (PRID) algorithm is presented. We start from the assumption that ambiguity is implicit in the classical formulation of the re-identification problem, as a specific individual may resemble one or more subjects by the color of dresses or the shape of the body. Therefore, we propose the introduction of the AMbiguity rAte in REidentification (AMARE) approach, which relates the results of a classical PRID pipeline on a specific dataset with their effectiveness in re-identification terms, exploiting the ambiguity rate (AR). As a consequence, the cumulative matching curves (CMC) used to show the results of a PRID algorithm will be filtered according to the AR. The proposed method gives a different interpretation of the output of PRID algorithms, because the CMC curves are processed, split and studied separately. Real experiments demonstrate that the separation of the results is really helpful in order to better understand the capabilities of a PRID algorithm.

Third generation surveillance systems are largely requested for intelligent surveillance of different scenarios such as public areas, urban traffic control, smart homes and so on. They are based on multiple cameras and processing modules that integrate data coming from a large surveillance space. The semantic interpretation of data from a multi-view context is a challenging task and requires the development of image processing methodologies that could support applications in extensive and real-time contexts. This paper presents a survey of automatic event detection functionalities that have been developed for third generation surveillance systems with a particular emphasis on open problems that limit the application of computer vision methodologies to commercial multi-camera systems.

In this paper, we propose an embedded vision system based on laser profilometry able to get the pose of a vehicle and its relative displacements with reference to the constitutive media of a structured environment. Fundamental equations for laser triangulation are developed and encoded for their actual implementation on an embedded system. It is made of a laser source that projects a line-shaped beam onto the environment and an on-chip camera able to frame the laser light. Images are then sent to the inexpensive Raspberry Pi onboard computer, which is responsible for processing tasks. For the first time, laser profilometry is coupled with the correlation of laser signatures on a low-cost and low-resource processing board for vehicle localization purposes. Several validation tests of the proposed sensor have proven the effectiveness of the system with respect to commercially available sensors such as inductive sensors and standard odometers, which fail when the vehicle crosses path interceptions or its wheels undergo unavoidable slippages. Moreover, further comparisons with other vision-based techniques have also proven the good performances of this embedded system for real-time localization of vehicles.

Service robots are expected to be used in many household in the near future, provided that proper interfaces are developed for the human robot interaction. Gesture recognition has been recognized as a natural way for the communication especially for elder or impaired people. With the developments of new technologies and the large availability of inexpensive depth sensors, real time gesture recognition has been faced by using depth information and avoiding the limitations due to complex background and lighting situations. In this paper the Kinect Depth Camera, and the OpenNI framework have been used to obtain real time tracking of human skeleton. Then, robust and significant features have been selected to get rid of unrelated features and decrease the computational costs. These features are fed to a set of Neural Network Classifiers that recognize ten different gestures. Several experiments demonstrate that the proposed method works effectively. Real time tests prove the robustness of the method for realization of human robot interfaces.

People tracking is a central and crucial point for the development of intelligent surveillance systems. When multiple cameras are used, the problem becomes more challenging as people re-identification is needed. Humans can greatly change their appearance according to posture, clothing and lighting conditions, thus defining features that describe people moving in large scenarios is a complex task. In this paper the problem of people re-identification and tracking is reviewed. The most used methodologies are discussed and insight into open problems and future research directions is provided.

Human action recognition is an important research area in the field of computer vision having a great number of real-world applications. This paper presents a multi-view action recognition framework that extracts human silhouette clues from different cameras, analyzes scene dynamics and interprets human behaviors by the integration of multivariate data in fuzzy rule-based system. Different features have been considered for the player action recognition some of them concerning the human silhouette analysis, and some others related to the ball and player kinematics. Experiments were carried out on a multi view image sequences of a public soccer data set.

This paper presents a mosaic based approach for enlarged view soccer video production that can be provided to the audience as a complementary view for greater enjoyment of relevant events, such as offside, counter attack or goal, that spread out all over the playing field. Firstly, an enlarged view of the whole field is produced by fusing the images of six cameras placed on the two sides of the field. Then a color transformation is applied to have uniform colors on the parts of the playing field acquired from different cameras. Finally, the players are segmented by each camera and projected onto the enlarged view to produce videos of the most interesting events.

Abstract:In the last years, smart surveillance has been one of the most active research topics in computer vision because of the wide spectrum of promising applications. Its main point is about the use of automatic video analysis technologies for surveillance purposes. In general, a processing framework for smart surveillance consists of a preliminary motion detection step in combination with high‐level reasoning that allows automatic understanding of evolutions of observed scenes. In this paper, we propose a surveillance framework based on a set of reliable visual algorithms that perform different tasks: a motion analysis approach that segments foreground regions is followed by three procedures, which perform object tracking, homographic transformations and edge matching, in order to achieve the real‐time monitoring of forbidden areas and the detection of abandoned or removed objects. Several experiments have been performed on different real image sequences acquired from a Messapic museum (indoor context) and the nearby archaeological site (outdoor context) to demonstrate the effectiveness and the flexibility of the proposed approach.

In the last decade, soccer video analysis has received a lot of attention from the scientific community. This increasing interest is motivated by the possible applications over a wide spectrum of topics: indexing, summarization, video enhancement, team and players statistics, tactics analysis, referee support, etc. The application of computer vision methodologies in the soccer context requires many problems to be faced: ball and players have to be detected in the images in any light and weather condition, they have to be localized in the field, tracked over time and finally their interactions have to be detected and analyzed. The latter task is fundamental, especially for statistic and referee decision support purposes, but, unfortunately, it has not received adequate attention from the scientific community and a lot of research remains to be done. In this paper a multicamera system is presented to detect the ball player interactions during soccer matches. The proposed method extracts, by triangulation from multiple cameras, the 3D ball and player trajectories and, by estimating the trajectory intersections, detects the ball-player interactions. An inference process is then introduced to determine the player kicking the ball and to estimate the interaction frame. The system was tested during several matches of the Italian first division football championship and experimental results demonstrated that the proposed method is robust and accurate.

The problem of ground truth generation is fundamental for many approaches of computer vision and image processing. In order to test algorithms for object segmentation, object tracking, object interactions, it is necessary to have image sequences in which the ground truth is determined in an objective way. In the context of visual surveillance where many people moves in the scene occluding each other, it could be very complex and hard the work of generating for each image the position of all the moving objects and maintain this information for all the period in which they remain in the scene. In this paper we propose a semi-automatic system that generates an initial ground truth estimation, and then provides a user-friendly interface to manually validate or correct the track results. The proposed system has been tested on some soccer video sequences that have been published on-line for being available to the scientific community, but it can be used also in other surveillance contexts.

In this paper, we investigate on the feasibility of multiple camera system for automatic offside detection. We propose six fixed cameras, properly placed on the two sides of the soccer field (three for each side) to reduce perspective and occlusion errors. The images acquired by the synchronized cameras are processed to detect the players' position and the ball position in real-time; a multiple view analysis is carried out to evaluate the offside event, considering the position of all the players in the field, determining the players who passed the ball, and determining if active offside condition occurred. The whole system has been validated using real-time images acquired during official soccer matches, and quantitative results on the system accuracy were obtained comparing the system responses with the ground truth data generated manually on a number of extracted significant sequences.

In this paper we present a people tracking algorithm which is able to detect and track soccer players in complex situations with varying light conditions, high frame rate, and real time processing. Object segmentation is performed by means of an algorithm based on background subtraction. In order to cope with presence of moving objects and light changes during the background modeling phase, an approach based on the evaluation of pixels energy content has been developed. Detected objects are then classified by means of an unsupervised clustering algorithm that allows the solution of blobs splitting and merging problems. For people tracking purpose we propose a stochastic approach based on the evaluation of the maximum a posteriori probability(MAP). First of all the algorithm evaluates geometrical information on the blob overlapping and then applies a color feature classification to track players and solve blob merging situations. Experimental tests have been carried out on long soccer image sequences in different weather and light conditions.

In this paper, we present a mobile robot global positioning system which combines passive RFID and monocular vision. The method is intended to solve the so-called kidnapped robot problem, that is, the problem of estimating the robot pose when it is first booted up or it has been kidnapped and taken to some unknown location. The main novelty of the approach is the use of fuzzy logic to cope with both uncertainty in RFID data and missing tag position information. Fuzzy logic allows us to manage inaccurate sensor data using rules easily understandable by humans. An additional advantage of the proposed method is that it does not require the robot to move around to solve ambiguities, nor does it need several tags to be properly arranged. Conversely, only one passive RFID tag and a single visual landmark are sufficient to univocally estimate the pose of the robot in the environment.

In this paper, a new method for a robust and efficient analysis of video sequences is presented; it allows the extraction of foreground objects and the classification of static foreground regions as abandoned or removed objects (ghosts). As a first step, the moving regions in the scene are detected by subtracting to the current frame a background model continuously adapted. Then, a shadow removing algorithm is used to extract the real shape of detected objects. Finally, moving objects are classified as abandoned or removed by matching the boundaries of static foreground regions. The method was successfully tested on both real image sequences acquired in our laboratory and some sequences from the PETS 2006 Datasets.

This paper describes a peculiar methodology for identifying and classifying the human body structures depicted in digital images. The methodology is articulated in a number of successive steps: an initial process of motion detection recognises human targets into the examined scene; subsequently, a synthetic representation of the previously detected human figures is obtained in form of skeleton, by means of a thinning operator. Finally, the pose estimation step is performed on the basis of a geometrical analysis conducted over a set of features related to the extracted skeletons. Experimental evidence of the appropriateness of the overall methodology is also presented.

Surveillance systems that automatically detect illegal behaviors performed by people unaware of the camera have a wide range of applications: security, healthcare, conservation of cultural heritage and so on. In particular the monitoring of museum and archaeological sites is one of the most challenging problems to be solved in order to avoid irreparable damages to historical heritage. In this paper a system able to check by common digital RGB cameras unexpected accesses to forbidden areas in a public museum is presented. The reliability of the proposed framework is shown by many experimental tests performed in the Messapic Museum of Egnathia (Italy).

In this paper, we introduce a new feature representation method for face recognition. The proposed method, referred as kernel ICA, combines the strengths of the kernel and independent component analysis (ICA) approaches. For performing kernel ICA, we employ an algorithm developed by F. R. Bach and M. I. Jordan. This algorithm has proven successful for separating randomly mixed auditory signals, but it has never been applied on bidimensional signals such as images. We compare the performance of kernel ICA with classical algorithms such as PCA and ICA within the context of appearance-based face recognition problem using the FERET and ORL databases. Experimental results show that both kernel ICA and ICA representations are superior to representations based on PCA for recognizing faces across days and changes in expressions.

In this work the complex behavior of localizing a mobile vehicle with respect to the door of the environment and then reaching the door has been developed. The robot uses visual information to detect and recognize the door and to determine its state with respect to it. This complex task has been divided into two separate behaviors: door-recognition and door-reaching. A supervised methodology based on learning by components has been applied for recognizing the door. Learning by components allows to recognize the door also in difficult situations such as partial occlusions and besides, it makes recognition independent of viewpoint variations and scale changes. An unsupervised methodology based on reinforcement learning has been used for the door-reaching behavior, instead. The image of the door gives information about the relative position of the vehicle with respect to the door. Then the Q-learning algorithm is used to generate the optimal state-action associations. The problem of defining the state and the action sets has been addressed with the aim of producing smooth paths, of reducing the effects of visual errors during real navigation, and of keeping low the computational cost during the learning phase. A novel way to obtain a continuous action set has been introduced: it uses a fuzzy model to evaluate the system state. Experimental results in real environment show both the robustness of the door-recognition behavior and the generality of the door-reaching behavior.

This paper presents an omnidirectional distributed vision system that learns to navigate a robot in an office-like environment without any knowledge about the calibration of the cameras or the robot control law. The system is composed of several omnidirectional vision agents (implemented with an omnidirectional camera and a computer). The first vision agent learns to control the robot with SARSA(/spl lambda/) reinforcement learning, using the LEM strategy to speed-up learning. Once the first vision agent learnt the correct policy, it transfers its knowledge to the other vision agents. The other vision agents might have different intrinsic and extrinsic camera parameters (that are unknown), so a certain amount of re-learning is needed. Reinforcement learning is well suited for this. In this paper, we present the structure of the learning system and the discussion about the optimal values for the learning parameters. During the experimentation the learning phase of the first agent has been carried out, then the knowledge propagation and the re-learning stage of three different agents have been tested. The experimental results demonstrate the feasibility of the approach and the possibility to port the system on the actual robot and cameras.

The Q-learning algorithm, for its simplicity and well-developed theory, has been largely used in the last years in order to realize different behaviors for autonomous vehicles. The most frequent applications required the standard tabular formulation with discrete sets of state and action. In order to consider continues variables, function approximators such as neural networks are required. In this work we investigate the neural approach of Q-learning on the robot navigation task of wall following. Some issues have been addressed in order to deal with the convergence problem and the need of huge training sets. The experience replay paradigm has been also applied to reduce the unlearning problem. Two different neural network architectures, which use different spatial decompositions of the sensory input, have been compared. The aim is to investigate how different choices of architecture can affect the learning convergence, the optimality of the final controller and the generalization ability.

This paper presents a vision-based technique for detecting targets of the environment which have to be reached by an autonomous mobile robot during its navigational tasks. The targets the robot has to reach are the doors of the authors' office building. The detection of the door has been performed by detecting its most significant components in the image and it is based on data classification. Two neural classifiers have been trained for recognizing single components of the door. Then a combining algorithm, based on heuristic considerations, checks that they are in the proper geometric configuration of the structure of the door. The novelty of this work is to use together colour and shape information for identifying features and for detecting the components of the target. The approach, based on learning by components, is able to cleverly solve the problems of scale changes, perspective variations and partial occlusions. The obtained detecting system has been tested on a large test set of real images showing a high reliability and robustness: doors of different rooms, under different illumination conditions and by different viewpoints have been successfully recognized. Results in terms of door detection rate and false positive rate are presented throughout the paper.

A problem related to the use of reinforcement learning (RL) algorithms on real robot applications is the difficulty of measuring the learning level reached after some experience. Among the different RL algorithms, the Q-learning is the most widely used in accomplishing robotic tasks. The aim of this work is to a priori evaluate the optimal Q-values for problems where it is possible to compute the distance between the current state and the goal state of the system. Starting from the Q-learning updating formula the equations for the maximum Q-weights, for optimal and non-optimal actions, have been computed considering delayed and immediate rewards. Deterministic and non deterministic grid-world environments have been also considered to test in simulations the obtained equations. Besides the convergence rates of the Q-learning algorithm have been compared using different learning rate parameters.

We present a learning algorithm that realizes a simple goal-reaching task for an autonomous vehicle when only visual information about the goal is provided. The robot has to reach a door from every position of the environment. The state of the system is based on visual information received by a TV camera placed on the mobile robot. The vision algorithm is able to determine the relative position between the vehicle and the door according to the slopes of the contour lines of the door. A learning phase is carried out in simulation to obtain the optimal state-action rules. The learned knowledge is then transferred on the real robot for the testing phase. Experimental results show the generality of the knowledge learned as the real robot is always able to execute its paths towards the door in different environments.

We describe a navigation approach, based on a reinforcement learning algorithm, that allows a mobile robot to move in an unknown indoor environment learning autonomously in a few trials the actions for reaching a particular goal location. The control architecture merges visual information and sonar readings to evaluate the state of the system to which the learning algorithm relates the best movement for reaching the goal. As a result, after limited experience, the robot learns efficient behavioral sequences and improves its learning incrementally. In addition the learning system adapts well to new situations of the environment or new task requirements. The results obtained in simulation and in real experiments, carried out in our laboratory, have shown that our system is tolerant to noise in sensor measurements and during each trial is always able to reach the goal.

The development of new techniques in sequencing nuclei acids has produced a great amount of sequence data and has led to the discovery of new relationships. In this paper, we study a method for parallelizing the algorithm WORDUP, which detects the presence of statistically significant patterns in DNA sequences. WORDUP implements an efficient method to identify the presence of statistically significant oligomers in a non-homologous group of sequences. It is based on a modified version of the Boyer-Moore algorithm, which is one of the fastest algorithms for string matching available in the literature. The aim of the parallel version of WORDUP presented here is to speed up the computational time and allow the analysis of a greater set of longer nucleotide sequences, which is usually impractical with sequential algorithms.

This paper is concerned with the probleni of determining the position of the vehicle both in goal-oriented and free- path navigations. A vision system for self location of a mobile robot has been devel- oped. By means of few known landmarks, the system has the ability of continuously monitoring position and compare these es- timates with the measures provided by the odometers. Besides a procedure has been im- plemented that recovers the position of the vehicle when the other sensors completely fail.

Presents an approach to detect moving obstacles in order to define a collision zone around them. The approach uses a camera system to provide image sequences from which the dynamic aspect of the scene can be recovered. In fact, if the camera samples the observation space at some time intervals, a two-dimensional image sequence may be created, due to the relative motion. The analysis of these sequences can be useful to recovery information about the relative notion between a mobile vehicle and the navigation space. The appearance of an obstacle can be recovered using optical flow analysis. The authors compute optical flow maps to solve the problem of detecting discontinuities in the flow field in order to segment the image in different moving objects. An estimation of depth values for the segmented points is accomplished by means of optical flow and focus of expansion. Experimental results on image sequences of a real scene are presented.>

Recovering three-dimensional visible surfaces is an important cue in computer vision. Like most of the inverse problems it is hard to be solved due to the insufficient and inaccurate data that can be collected using passive sensors. Data fusion and integration over time can be used to overcome this problem. In this paper the depth and orientation are acquired from a scene and are used together to build a dense map of the visible surface for each of several points of view. An incremental estimator is used to integrate these maps, as soon as they become available, in a more reliable result.>