Recent years have witnessed a growing interest in lower limb exoskeletons (LLEs) designed to restore walking capabilities for individuals with motor impairments. Currently, the research community faces the challenge of enhancing the translational impact of this technology by proposing solutions to move from laboratory and clinical settings to daily life scenarios. In this context, the topic of environment-adaptive gait planning (EAGP) is of particular interest and importance. EAGP refers to the ability of LLEs to assist in walking on various terrains (e.g., stairs and slopes) and under different conditions (e.g., the presence of obstacles and uneven ground) by adapting their behavior according to the context. However, the proliferation of publications has led to a general lack of convergence within the scientific community toward a common definition of this research topic. Therefore, this article proposes a review of EAGP for LLEs. The objectives are to: 1) provide a clear definition of EAGP, highlighting its main components and characteristics; 2) propose a taxonomy for classifying the most frequently used techniques in the state of the art; and 3) conduct a comprehensive survey of the literature, situating each article within the proposed taxonomy. In addition, we identify the most urgent future challenges toward the development of more autonomous LLEs.

Covert visuospatial attention (CVSA) enables users to direct focus without eye movements, offering a promising non-invasive control modality for brain-machine interface (BMI) systems. Traditional CVSA-based BMIs rely on decoding α-band electroencephalography (EEG) activity from whole-trial data, often overlooking the temporal dynamics of attentional processing. In this study, we propose a novel control framework that explicitly separates the CVSA task into two subcomponents: shift attention and sustained attention. EEG signals were recorded from nine participants performing a CVSA task in a real-world environment. Through spectral feature analysis and temporal segmentation, we identify distinct patterns associated with each subcomponent and train dedicated classifiers accordingly. Our pseudo-online analysis suggests that combining the outputs of two task-specific classifiers significantly improves both trial-level classification accuracy (by an average of +23.74%) and area under the curve (AUC) (by +4.49%) compared to the conventional approach. These findings suggest that modelling the temporal structure of CVSA can enhance decoding performance and offer a more robust foundation for attention-based BMI.

Motor imagery (MI) is a fundamental brain-machine interface (BMI) paradigm in which users learn how to modulate their brain signals to voluntarily and selectively activate specific areas of the sensorimotor cortex. The self-paced nature of this kinesthetic imagination makes MI well-suited for several human-robot interaction (HRI) scenarios. However, the performance of MI decoding is highly dependent on both the user’s expertise and the quality of the decoding algorithm. To address these challenges, we propose a method that integrates environmental data from robotic sensors to enhance the MI decoding process. Preliminary experiments indicate that this approach improves decoding accuracy and the overall performance of the brain-driven system, opening new opportunities for research on how machines can enhance the usability of BMIs systems.

Despite the importance of human-robot collaborative transportation of flexible material in many industrial scenarios, many works in the literature assume a passive role for the robot during the collaboration. The robot can only follow the human partner, without providing assistance in the more challenging phase of the collaboration such as precise material positioning. This work presents a framework for co-transportation, proposing a distance-based policy for dynamic leader role allocation through the task. For large distances from the target pose, the robot is mainly controlled by vision-based manual guidance exploiting haptic feedback and 3D human pose information; instead, close to the target material position, the robot acts as a leader guiding the human operator. The proposed framework is evaluated considering a carbon fiber draping task, which requires both co-transportation and precise positioning of flexible materials. Experimental results demonstrate how the robot leading the task in the final stage allows to achieve high task efficiency and alleviates human stress in the execution of the task.

Panoramic cameras offer a 4π steradian field of view, which is desirable for tasks like people detection and tracking since nobody can exit the field of view. Despite the recent diffusion of low-cost panoramic cameras, their usage in robotics remains constrained by the limited availability of datasets featuring annotations in the robot space, including people’s 2D or 3D positions. To tackle this issue, we introduce PanNote, an automatic annotation tool for people’s positions in panoramic videos. Our tool is designed to be cost-effective and straightforward to use without requiring human intervention during the labeling process and enabling the training of machine learning models with low effort. The proposed method introduces a calibration model and a data association algorithm to fuse data from panoramic images and 2D LiDAR readings. We validate the capabilities of PanNote by collecting a real-world dataset. On these data, we compared manual labels, automatic labels and the predictions of a baseline deep neural network. Results clearly show the advantage of using our method, with a 15-fold speed up in labeling time and a considerable gain in performance while training deep neural models on automatically labelled data.

Robotic waste sorting poses significant challenges in both perception and manipulation, given the extreme variability of objects that should be recognized on a cluttered conveyor belt. While deep learning has proven effective in solving complex tasks, the necessity for extensive data collection and labeling limits its applicability in real-world scenarios like waste sorting. To tackle this issue, we introduce a data augmentation method based on a novel GAN architecture called wasteGAN. The proposed method allows to increase the performance of semantic segmentation models, starting from a very limited bunch of labeled examples, such as few as 100. The key innovations of wasteGAN include a novel loss function, a novel activation function, and a larger generator block. Overall, such innovations helps the network to learn from limited number of examples and synthesize data that better mirrors real-world distributions. We then leverage the higher-quality segmentation masks predicted from models trained on the wasteGAN synthetic data to compute semantic-aware grasp poses, enabling a robotic arm to effectively recognizing contaminants and separating waste in a real-world scenario. Through comprehensive evaluation encompassing dataset-based assessments and real-world experiments, our methodology demonstrated promising potential for robotic waste sorting, yielding performance gains of up to 5.8% in picking contaminants. The project page is available at https://github.com/bach05/wasteGAN.git.

Semantic segmentation of point clouds is an essential task for understanding the environment in autonomous driving and robotics. Recent range-based works achieve real-time efficiency, while point- and voxel-based methods produce better results but are affected by high computational complexity. Moreover, highly complex deep learning models are often not suited to efficiently learn from small datasets. Their generalization capabilities can easily be driven by the abundance of data rather than the architecture design. In this paper, we harness the information from the three-dimensional representation to proficiently capture local features, while introducing the range image representation to incorporate additional information and facilitate fast computation. A GPU-based KDTree allows for rapid building, querying, and enhancing projection with straightforward operations. Extensive experiments on SemanticKITTI and nuScenes datasets demonstrate the benefits of our modification in a "small data" setup, in which only one sequence of the dataset is used to train the models, but also in the conventional setup, where all sequences except one are used for training. We show that a reduced version of our model not only demonstrates strong competitiveness against full-scale state-of-the-art models but also operates in real-time, making it a viable choice for real-world case applications. The code of our method is available at https://github.com/Bender97/WaffleAndRange.

Powered lower limb exoskeletons (LLEs) have emerged as wearable robots designed to augment users’ locomotion capabilities, offering mechanical support and additional power for both healthy and impaired subjects. However, current assistive exoskeletons are limited by predefined motion trajectories, hindering adaptability to unstructured environments encountered in daily life. To address this limitation, this paper proposes an environment-adaptive gait planning (EAGP) solution. The approach integrates scene understanding, pose estimation, and adaptive gait planning modules. A novel Collision-Free Foot Trajectory Generator (CFFTG) algorithm facilitates obstacle avoidance by computing collision-free foot trajectories, enhancing safety and adaptability. Through inverse kinematics, the planned trajectories are converted into angular joint trajectories for execution by low-level control. This comprehensive framework aims to enhance the adaptability and safety of LLEs, paving the way for broader real-world applications beyond clinical and research settings.

Robot grasping has been widely studied in the last decade. Recently, Deep Learning made possible to achieve remarkable results in grasp pose estimation, using depth and RGB images. However, only few works consider the choice of the object to grasp. Moreover, they require a huge amount of data for generalizing to unseen object categories. For this reason, we introduce the Few-shot Semantic Grasping task where the objective is inferring a correct grasp given only five labelled images of a target unseen object. We propose a new deep learning architecture able to solve the aforementioned problem, leveraging on a Few-shot Semantic Segmentation module. We have evaluated the proposed model both in the Graspnet dataset and in a real scenario. In Graspnet, we achieve 40,95% accuracy in the Few-shot Semantic Grasping task, outperforming baseline approaches. In the real experiments, the results confirmed the generalization ability of the network.

Hand-eye calibration is the problem of estimating the spatial transformation between a reference frame, usually the base of a robot arm or its gripper, and the reference frame of one or multiple cameras. Generally, this calibration is solved as a non-linear optimization problem, what instead is rarely done is to exploit the underlying graph structure of the problem itself. Actually, the problem of hand-eye calibration can be seen as an instance of the Simultaneous Localization and Mapping (SLAM) problem. Inspired by this fact, in this work we present a pose-graph approach to the hand-eye calibration problem that extends a recent state-of-the-art solution in two different ways: i) by formulating the solution to eye-on-base setups with one camera; ii) by covering multi-camera robotic setups. The proposed approach has been validated in simulation against standard hand-eye calibration methods. Moreover, a real application is shown. In both scenarios, the proposed approach overcomes all alternative methods. We release with this paper an open-source implementation of our graph-based optimization framework for multi-camera setups.

This paper presents Hi-ROS (Human Interaction in ROS), an open source framework focused on real-time accurate assessment of human motion. The system offers a series of tools to track multiple people in real-time by exploiting a calibrated camera network. No assumptions are made about the typology or number of cameras, nor about the body pose estimation algorithm used to extract the 3D poses of the people in the scene. The tools provided by Hi-ROS include a Skeleton Tracker to ensure temporal consistency of the detected poses, a Skeleton Merger to fuse the tracks from multiple cameras, thus limiting flickering phenomena, a Skeleton Optimizer to ensure limb length consistency, and a Skeleton Filter to perform real-time smoothing of the detected joint trajectories. Accuracy, tracking robustness, and real-time performance of the proposed system were evaluated on a public dataset, containing both single-person and multi-person sequences with up to 4 people interacting. The results obtained using different subsets of the proposed tools show how the complete Hi-ROS pipeline provides accurate and reliable estimates also in challenging scenarios, with a reduction of the RMSE of up to 27 % with respect to a pure tracking approach. This work aims to push forward the development of unobtrusive human–robot interaction applications, multi-person automated posture analyses, rehabilitation performance assessments, and any possible application enabled by real-time accurate assessment of human motion via markerless motion capture.

In this paper, we present a control system for the continuous teleoperation of a robotic manipulator via brain-machine interface (BMI). The proposed solution is based on shared control approach that allows the user to only focus on the operational tasks, while the low-level control details are automatically handled by the robotic intelligence. The user drives the manipulator through the imagination of limb movements (both hands vs. both feet) and a parameterized mapping function is implemented to convert the continuous BMI outputs into robot velocity commands which are sent to the shared control framework. The latter consists in: (i) a target predictor module, to infer the most probable target objects from the sequence of BMI commands; (ii) a control module based on an improved version of artificial potential fields (APF) to assist the user in reaching the target while avoiding collisions with obstacles in the environment. The system has been tested with a sample subject in a tabletop reach-to-grasp experiment with multiple target objects and obstacles achieving a success rate of 80%. The proposed system could be used in the future to help people with severe motor disabilities in performing daily life operations, such as drinking, feeding or manipulating objects.

This article proposes a novelshared intelligencesystem for brain-machine interface (BMI) teleoperated mobile robots where user’s intention and robot’s intelligence are concurrent elements equally participating in the decision process. We designed the system to rely onpoliciesguiding the robot’s behavior according to the current situation. We hypothesized that the fusion of thesepolicieswould lead to the identification of the next, most probable, location of the robot in accordance with the user’s expectations. We asked 13 healthy subjects to evaluate the system during teleoperated navigation tasks in a crowded office environment with a keyboard (reliable interface) and with 2-class motor imagery (MI) BMI (uncertain control channel). Experimental results show that ourshared intelligencesystem 1) allows users to efficiently teleoperate the robot in both control modalities; 2) it ensures a level of BMI navigation performances comparable to the keyboard control; 3) it actively assists BMI users in accomplishing the tasks. These results highlight the importance of investigating advanced human-machine interaction (HMI) strategies and introducing robotic intelligence to improve the performances of BMI actuated devices.

In shared control teleoperation, the robot assists the user in accomplishing the desired task. Rather than simply executing the user’s command, the robot attempts to integrate it with information from the environment, such as obstacle and/or goal locations, and it modifies its behavior accordingly. In this article, we propose a real-time shared control teleoperation framework based on an artificial potential field approach improved by the dynamic generation of escape points around the obstacles. These escape points are virtual attractive points in the potential field that the robot can follow to overcome the obstacles more easily. The selection of which escape point to follow is done in real time by solving a soft-constrained problem optimizing the reaching of the most probable goal, estimated from the user’s action. Our proposal has been extensively compared with two state-of-the-art approaches in a static cluttered environment and a dynamic setup with randomly moving objects. Experimental results showed the efficacy of our method in terms of quantitative and qualitative metrics. For example, it significantly decreases the time to complete the tasks and the user’s intervention, and it helps reduce the failure rate. Moreover, we received positive feedback from the users that tested our proposal. Finally, the proposed framework is compatible with both mobile and manipulator robots.

This review article discusses the definition and implementation of brain–computer interface (BCI) system relying on brain connectivity (BC) and machine learning/deep learning (DL) for motor imagery (MI)-based applications. During the past few years, many approaches have been explored in terms of types of neurological sources of information, feature extraction, and intention prediction for BCI applications. Two novel aspects are becoming increasingly interesting for the BCI community: BC modeling and DL. The former aims at describing the interactions among different brain regions as connectivity patterns that reflect the dynamics of information flow either at rest or when performing a task. The latter is becoming pervasive for its capability of modeling and predicting complex data, where a huge amount of information is involved. In this scenario, we conducted a systematic literature review on BCI studies that led to the selection of 34 articles meeting all the required criteria. This provides evidence of the rapid growth of the topic over the past few years, though being still in its infancy. The last part of this article is dedicated to this new frontier of BCI that we call MI BC-based computer interfaces highlighting the potential of BC features. This, jointly with DL as enabling technology, has the potential of improving the performance of electroencephalography-based systems.

In this work we present an industrial system for the inspection of composite parts in the aerospace industry, based on X-ray sensors and robotic manipulators. Such system is designed to identify any type of defects such as, missing gluing, core cell deformation, cracks or foreign objects, which may occur between layers of which these objects are composed. The inspection process involves back-projection of X-ray images onto the 3D CAD model of the inspected part, to directly locate the defects on the part itself. The complete system has been implemented in a real industrial workcell that involves two synchronized robots equipped with a X-ray source-detector system. The two robots move autonomously along a pre-computed trajectory without any human intervention, and the back-projection of the acquired images is efficiently performed at run-time using the proposed algorithm. The experiments demonstrate that the X-ray images back-projection is successful and can effectively replace standard manually guided inspections. This has a high impact on the factory automation cycle since it helps to reduce the effort and time needed for each inspection task. This work is part of a EU funded project called SPIRIT.

This paper describes a calibration method for RGB-D camera networks consisting of not only static overlapping, but also dynamic and non-overlapping cameras. The proposed method consists of two steps: online visual odometry-based calibration and depth image-based calibration refinement. It first estimates the transformations between overlapping cameras using fiducial tags, and bridges non-overlapping camera views through visual odometry that runs on a dynamic monocular camera. Parameters such as poses of the static cameras and tags, as well as dynamic camera trajectory, are estimated in the form of the pose graph-based online landmark SLAM. Then, depth-based ICP and floor constraints are added to the pose graph to compensate for the visual odometry error and refine the calibration result. The proposed method is validated through evaluation in simulated and real environments, and a person tracking experiment is conducted to demonstrate the data integration of static and dynamic cameras.

The P300 is an endogenous event-related potential (ERP) involved in several cognitive processes, apparently preserved between adults and children. In the pediatric age it shows different age-related characteristics. Its application in Brain Computer Interface (BCI) pediatric research remains to date still unclear. The aim of this paper is to give an overview of the maturational aspects of the visual P300, that could be used for developing BCI paradigms in the pediatric age.

This work proposes a general formulation for the Object Sorting problem, suitable to describe any non-deterministic environment characterized by friendly and adversarial interference. Such an approach, coupled with a Deep Reinforcement Learning algorithm, allows training policies to solve different sorting tasks without adjusting the architecture or modifying the learning method. Briefly, the environment is subdivided into a clutter, where objects are freely located, and a set of clusters, where objects should be placed according to predefined ordering and classification rules. A 3D grid discretizes such environment: the properties of an object within a cell depict its state. Such attributes include object category and order. A Markov Decision Process formulates the problem: at each time step, the state of the cells fully defines the environment's one. Users can custom-define object classes, ordering priorities, and failure rules. The latter by assigning a non-uniform risk probability to each cell. Performed experiments successfully trained and validated a Deep Reinforcement Learning model to solve several sorting tasks while minimizing the number of moves and failure probability. Obtained results demonstrate the capability of the system to handle non-deterministic events, like failures, and unpredictable external disturbances, like human user interventions.

This work proposes an overview of the recent applications of brain-computer interface (BCI) technology for pediatric populations. Current BCIs have demonstrated the possibility to provide an alternative communication and interaction channel for people suffering from severe motor disabilities. However, to date research has been predominantly conducted in adults, only a few systems have been applied to pediatric population. A survey was carried out to show the ongoing trends of using BCI systems with children. We discuss three areas of applications where BCI might be helpful to children - "Communication & Control", "BCI Gaming for Neurofeedback Training" and "Rehabilitation" - highlighting the current limitations and the possible future challenges.

The increasing interest of the research community in the intertwined fields of brain-machine interface (BMI) and robotics has led to the development of a variety of brain-actuated devices, ranging from powered wheelchairs and telepresence robots to wearable exoskeletons. Nevertheless, in most cases, the interaction between the two systems is still rudimentary, allowing only an unidirectional simple communication from the BMI to the robot that acts as a mere passive end-effector. This limitation could be due to the lack of a common research framework, facilitating the integration of these two technologies. In this scenario, we proposed ROS-Neuro to overcome the aforementioned limitations by providing a common middleware between BMI and robotics. In this work, we present a working example of the potentialities of ROS-Neuro by describing a full closed-loop implementation of a BMI based on motor imagination. The paper shows the general structure of a closed-loop BMI in ROS-Neuro and describes the specific implementation of the packages related to the proposed motor imagery BMI, already available online with source codes, tutorials and documentations. Furthermore, we show two practical case scenarios where the implemented BMI is used to control a computer game or a telepresence robot with ROS-Neuro. We evaluated the performance of ROS-Neuro by ensuring comparable results with respect to a previous BMI software already validated. Results demonstrated the correct behavior of the provided packages.

Recently, the complexity analysis of brain activity has shown the possibility to provide additional information to discriminate between rest and motion in real-time. In this work, we propose a novel entropy-based machine learning method to classify between standing and walking conditions from the sole brain activity. The Shannon entropy has been used as a complexity measure of electroencephalography (EEG) signals and subject-specific features for classification have been selected by Common Spatial Patterns (CSP) filter. Exploiting these features with a linear classifier, we achieved > 85% of classification accuracy over a long period (≈ 25 min) of standing and treadmill walking on 11 healthy subjects. Moreover, we implemented the proposed approach to successfully discriminate in real-time between standing and over-ground walking on one healthy subject. We suggest that the reliable discrimination of rest against walking conditions achieved by the proposed method may be exploited to have more stable control of devices to restore locomotion, avoiding unpredictable and dangerous behaviors due to the delivery of undesired control commands.

In this paper, we propose a ROS-based system to reconstruct the motion of human upper limb based on data collected with two Myo armbands in a hybrid manner. The inertial sensors' information are fused to reconstruct shoulder and elbow kinematics. Electromyographic (EMG) signals are used to estimate wrist kinematics, to fully capture the motion of the 5-DoF (degree of freedom) user's arm. The system shows a good pose estimation accuracy compared to the XSens suit with an average RMSE of 6.61°±3.31°and a R2of 0.90±0.07.

Brain-Computer Interface systems have been widely studied and explored with adults demonstrating the possibility to achieve augmentative communication and control directly from the users' brain. Nevertheless, the study and the exploitation of the BCI in children seems to be limited. In this paper we propose and present for the first time a Brain-Robot Interface enabling children to mentally drive a robot. With this regards, we exploit the combination of a P300-based Brain-Computer Interface and a shared-autonomy approach to achieve a reliable and safe robot navigation. We tested our system in a pilot study involving five children. Our preliminary results highlight the advantages of using an accumulation framework, thanks to which the performance of the children reached the 81.67 % ± 12.7 on average in terms of accuracy. During the experiments, the shared-autonomy approach involved a low-level intelligent control on board of the robot to avoid obstacles, enabling an effective navigation also with a small number of commands.

This paper presents a novel real-time tracking system capable of improving body pose estimation algorithms in distributed camera networks. The first stage of our approach introduces a linear Kalman filter operating at the body joints level, used to fuse single-view body poses coming from different detection nodes of the network and to ensure temporal consistency between them. The second stage, instead, refines the Kalman filter estimates by fitting a hierarchical model of the human body having constrained link sizes in order to ensure the physical consistency of the tracking. The effectiveness of the proposed approach is demonstrated through a broad experimental validation, performed on a set of sequences whose ground truth references are generated by a commercial marker-based motion capture system. The obtained results show how the proposed system outperforms the considered state-of-the-art approaches, granting accurate and reliable estimates. Moreover, the developed methodology constrains neither the number of persons to track, nor the number, position, synchronization, frame-rate, and manufacturer of the RGB-D cameras used. Finally, the real-time performances of the system are of paramount importance for a large number of real-world applications.

Recent advances in the Brain-Machine Interface (BMI) and in the robotic fields allowed researchers to design a new generation of brain-actuated neuroprostheses to control a variety of assistive devices-ranging from powered wheelchairs to telepresence robots and robotic arms. However, the current interaction between BMI and robotic applications is still at its infancy. One of the reasons for the limited integration between these two technologies might be identified in the lack of a common research framework. In this scenario, ROS-Neuro may represent a solution to this challenge by providing a common middleware between BMI and robotics.In this work, we propose two packages for the ROS-Neuro middleware to acquire neural signals from different commercial acquisition devices and to store them into common data formats. We describe the usage and the main functionalities of both packages such as the provided nodes, the published and subscribed topics and the available services. Furthermore, the packages are available online together with source codes, tutorials and documentations. We evaluated the performances of the two packages to ensure the absence of delays or missing data with three commercial acquisition systems and in different operating configurations. Results demonstrated the correct behavior of the provided packages. The acquisition and recorder packages represent the first releases in the ROS-Neuro ecosystem and they aim at boosting both the shared development and the research activity in the multidisciplinary field of brain-actuated robots.

The identification of intentionally delivered commands is a challenge in Brain Computer Interfaces (BCIs) based on Sensory-Motor Rhythms (SMR). It is of fundamental importance that BCI systems controlling a robotic device (i.e., upper limb prosthesis) are capable of detecting if the user is in the so called Intentional Non-Control (INC) state (i.e., holding the prosthesis in a given position). In this work, we propose a novel approach based on the entropy of the Electroencephalogram (EEG) signals to provide a continuous identification of motion intention. Results from ten healthy subjects suggest that the proposed system can be used for reliably predicting motion in real-time at a framerate of 8 Hz with 80%±5% of accuracy. Moreover, motion intention can be detected more than 1 second before muscular activation with an average accuracy of 76%±11%.

Applications that provide location related services need to understand the environment in which humans live such that verbal references and human interaction are possible. We formulate this semantic labelling task as the problem of learning the semantic labels from the perceived 3D structure. In this contribution we propose a batch approach and a novel multi-view frame fusion technique to exploit multiple views for improving the semantic labelling results. The batch approach works offline and is the direct application of an existing single-view method to scene reconstructions with multiple views. The multi-view frame fusion works in an incremental fashion accumulating the single-view results, hence allowing the online multi-view semantic segmentation of single frames and the offline reconstruction of semantic maps. Our experiments show the superiority of the approaches based on our fusion scheme, which leads to a more accurate semantic labelling.

This paper shows and evaluates a novel approach to integrate a non-invasive Brain-Computer Interface (BCI) with the Robot Operating System (ROS) to mentally drive a telepresence robot. Controlling a mobile device by using human brain signals might improve the quality of life of people suffering from severe physical disabilities or elderly people who cannot move anymore. Thus, the BCI user can actively interact with relatives and friends located in different rooms thanks to a video streaming connection to the robot. To facilitate the control of the robot via BCI, we explore new ROS-based algorithms for navigation and obstacle avoidance in order to make the system safer and more reliable. In this regard, the robot exploits two maps of the environment, one for localization and one for navigation, and both are used as additional visual feedback for the BCI user to control the robot position. Experimental results show a decrease of the number of commands needed to complete the navigation task, suggesting a reduction user's cognitive workload. The novelty of this work is to provide a first evidence of an integration between BCI and ROS that can simplify and foster the development of software for BCI driven robotics devices.

Color-depth cameras (RGB-D cameras) have become the primary sensors in most robotics systems, from service robotics to industrial robotics applications. Typical consumer-grade RGB-D cameras are provided with a coarse intrinsic and extrinsic calibration that generally does not meet the accuracy requirements needed by many robotics applications [e.g., highly accurate three-dimensional (3-D) environment reconstruction and mapping, high precision object recognition, localization, etc.]. In this paper, we propose a human-friendly, reliable, and accurate calibration framework that enables to easily estimate both the intrinsic and extrinsic parameters of a general color-depth sensor couple. Our approach is based on a novel two components error model. This model unifies the error sources of RGB-D pairs based on different technologies, such as structured-light 3-D cameras and time-of-flight cameras. Our method provides some important advantages compared to other state-of-the-art systems: It is general (i.e., well suited for different types of sensors), based on an easy and stable calibration protocol, provides a greater calibration accuracy, and has been implemented within the robot operating system robotics framework. We report detailed experimental validations and performance comparisons to support our statements.

This paper deals with the problem of detecting fallen people lying on the floor by means of a mobile robot equipped with a 3D depth sensor. In the proposed algorithm, inspired by semantic segmentation techniques, the 3D scene is over-segmented into small patches. Fallen people are then detected by means of two SVM classifiers: the first one labels each patch, while the second one captures the spatial relations between them. This novel approach showed to be robust and fast. Indeed, thanks to the use of small patches, fallen people in real cluttered scenes with objects side by side are correctly detected. Moreover, the algorithm can be executed on a mobile robot fitted with a standard laptop making it possible to exploit the 2D environmental map built by the robot and the multiple points of view obtained during the robot navigation. Additionally, this algorithm is robust to illumination changes since it does not rely on RGB data but on depth data. All the methods have been thoroughly validated on the IASLAB-RGBD Fallen Person Dataset, which is published online as a further contribution. It consists of several static and dynamic sequences with 15 different people and 2 different environments.

This paper presents a fast and robust multiple object tracking algorithm based on an RGB-D version of the MeanShift tracking algorithm and exploiting RGB-D camera networks when multiple RGB-D sensors are available. The original Mean-Shift algorithm has been improved in three ways. First, a color-depth Joint Probability Density Function is proposed for taking into account both depth and color information. Secondly, we propose an occlusion detection mechanism which can handle long-term occlusions even when objects move fast and unpredictably. Finally, when multiple views are available, we combine the tracking outcomes from all the RGB-D sensors in our network to deal with the identity confusion problem and enhance the overall tracking performance. Experimental results demonstrate that the proposed scheme is robust, realtime and has yielded a marked improvement with respect to the state-of-the-art in terms of tracking quality. As a further contribution, we released our work as open-source in order to provide the best benefit to the wide Computer Vision and Robotics community.

Microsoft Kinect had a key role in the development of consumer depth sensors being the device that brought depth acquisition to the mass market. Despite the success of this sensor, with the introduction of the second generation, Microsoft has completely changed the technology behind the sensor from structured light to Time-Of-Flight. This paper presents a comparison of the data provided by the first and second generation Kinect in order to explain the achievements that have been obtained with the switch of technology. After an accurate analysis of the accuracy of the two sensors under different conditions, two sample applications, i.e., 3D reconstruction and people tracking, are presented and used to compare the performance of the two sensors.

In this paper, an automatic optical inspection system for checking the sequence of colored wires in electric cable is presented. The system is able to inspect cables with flat connectors differing in the type and number of wires. This variability is managed in an automatic way by means of a self-learning subsystem and does not require manual input from the operator or loading new data to the machine. The system is coupled to a connector crimping machine and once the model of a correct cable is learned, it can automatically inspect each cable assembled by the machine. The main contributions of this paper are: (i) the self-learning system; (ii) a robust segmentation algorithm for extracting wires from images even if they are strongly bent and partially overlapped; and (iii) a color recognition algorithm able to cope with highlights and different finishing of the wire insulation. We report the system evaluation over a period of several months during the actual production of large batches of different cables; tests demonstrated a high level of accuracy and the absence of false negatives, which is a key point in order to guarantee defect-free productions.

The availability of affordable depth sensors in conjunction with common RGB cameras (even in the same device, e.g. the Microsoft Kinect) provides robots with a complete and instantaneous representation of both the appearance and the 3D structure of the current surrounding environment. This type of information enables robots to safely navigate, perceive and actively interact with other agents inside the working environment. It is clear that, in order to obtain a reliable and accurate representation, not only the intrinsic parameters of each sensors should be precisely calibrated, but also the extrinsic parameters relating the two sensors should be precisely known. In this paper, we propose a human-friendly and reliable calibration framework, that enables to easily estimate both the intrinsic and extrinsic parameters of a camera-depth sensor couple. Real world experiments using a Kinect show improvements for both the 3D structure estimation and the association tasks.

In this work, we describe a novel method for creating 3D models of persons freely moving in front of a consumer depth sensor and we show how they can be used for long-term person re-identification. For overcoming the problem of the different poses a person can assume, we exploit the information provided by skeletal tracking algorithms for warping every point cloud frame to a standard pose in real time. Then, the warped point clouds are merged together to compose the model. Re-identification is performed by matching body shapes in terms of whole point clouds warped to a standard pose with the described method. We compare this technique with a classification method based on a descriptor of skeleton features and with a mixed approach which exploits both skeleton and shape features. We report experiments on two datasets we acquired for RGB-D re-identification which use different skeletal tracking algorithms and which are made publicly available to foster research in this new research branch.

In this paper we propose a novel methodology for people re-identification based on skeletal information. Features are evaluated on the skeleton joints and a highly distinctive and compact feature-based signature is generated for each user by concatenating descriptors of all visible joints. We compared a number of state-of-the-art 2D and 3D feature descriptors to be used with our signature on two newly acquired public datasets for people re-identification with RGB-D sensors. Moreover, we tested our approach against the best re-identification methods in the literature and on a widely used public video surveillance dataset. Our approach proved to be robust to strong illumination changes and occlusions. It achieved very high performance also on low resolution images, overcoming state-of-the-art methods in terms of recognition accuracy and efficiency. These features make our approach particularly suited for mobile robotics.

Motion sensors as inertial measurement units (IMU) are widely used in robotics, for instance in the navigation and mapping tasks. Nowadays, many low cost micro electro mechanical systems (MEMS) based IMU are available off the shelf, while smartphones and similar devices are almost always equipped with low-cost embedded IMU sensors. Nevertheless, low cost IMUs are affected by systematic error given by imprecise scaling factors and axes misalignments that decrease accuracy in the position and attitudes estimation. In this paper, we propose a robust and easy to implement method to calibrate an IMU without any external equipment. The procedure is based on a multi-position scheme, providing scale and misalignments factors for both the accelerometers and gyroscopes triads, while estimating the sensor biases. Our method only requires the sensor to be moved by hand and placed in a set of different, static positions (attitudes). We describe a robust and quick calibration protocol that exploits an effective parameterless static filter to reliably detect the static intervals in the sensor measurements, where we assume local stability of the gravity's magnitude and stable temperature. We first calibrate the accelerometers triad taking measurement samples in the static intervals. We then exploit these results to calibrate the gyroscopes, employing a robust numerical integration technique. The performances of the proposed calibration technique has been successfully evaluated via extensive simulations and real experiments with a commercial IMU provided with a calibration certificate as reference data.

This paper proposes a very fast and robust multi-people tracking algorithm suitable for mobile platforms equipped with a RGB-D sensor. Our approach features a novel depth-based sub-clustering method explicitly designed for detecting people within groups or near the background and a three-term joint likelihood for limiting drifts and ID switches. Moreover, an online learned appearance classifier is proposed, that robustly specializes on a track while using the other detections as negative examples. Tests have been performed with data acquired from a mobile robot in indoor environments and on a publicly available dataset acquired with three RGB-D sensors and results have been evaluated with the CLEAR MOT metrics. Our method reaches near state of the art performance and very high frame rates in our distributed ROS-based CPU implementation.

This paper presents a system for checking connectors while cables are being crimped to them. The system verifies that the wires color sequence is correct: an accurate color analysis technique has then been developed, in order to discriminate between similar colors, and filter noise factors.

In this work, we propose a robust and efficient method to build dense 3D maps, using only the images grabbed by an omnidirectional camera. The map contains exhaustive information about both the structure and the appearance of the environment and it is well suited also for large scale environments. We start from the assumption that the surrounding environment (the scene) forms a piecewise smooth surface represented by a triangle mesh. Our system is able to infer, without any odometry information, the structure of the environment along with the ego-motion of the camera by performing a robust tracking of the projection of this surface in the omnidirectional image. The key idea is to use a guess of the triangle mesh subdivision based on a constrained Delaunay triangulation built according to a set of point features and edgelet features extracted from the image. In such a way, we take into account both the corners and the edges of the scene imaged by the camera, constrained by the topology of the triangulation in order to improve the stability of the tracking process. Both motion and structure parameters are estimated using a direct method inside an optimization framework, taking into account the topology of the subdivision in a robust and efficient way. We successfully tested our system in a challenging urban scenario along a large loop using an omnidirectional camera mounted on the roof of a car.

Touch is an important means for communication among humans. Sport instructors or dance teachers often use touch to adjust students' postures in a very intuitive way. Using tactile instructions appears thus to be a very appealing modality for developing humanoid robot motions as well. Spontaneous interpretation of tactile instructions given by users reveals itself to be a complex task for artificial systems. This paper describes a proof of concept system for robot motion creation based on tactile interaction. The system is interesting for two reasons. Firstly, it shows the feasibility of using tactile instructions for motion development. Secondly, it can be used as a tool for studying the way humans intuitively use touch to communicate. This, in turn, will allow the development of better algorithms for predicting the meaning of tactile instructions. Results of a pilot experiment are discussed, and a first set of features of tactile communication, yielded by the analysis of the data collected, is identified.

This paper describes a sensor for autonomous surveillance capable of continuously monitoring the environment, while acquiring detailed images of specific areas. This is achieved by exploiting an omnidirectional camera and a PTZ camera, assembled together on a single mount. The two cameras form a single vision sensor, since data obtained processing the two images are used in a cooperative way. This system solves the problem affecting systems based on PTZ cameras only, since it does not exist a tracking system working reliably on PTZ images: the problem is solved here by performing the tracking in the omnidirectional image. This vision sensor is used in a surveillance application, that detects moving objects, and records all the faces of the people walking close to the sensor. It could be used to navigate or instruct a security or a service mobile robot.

Brain computer interface is a system that offers also a support to the patients with neuromuscular diseases as amyotrophic lateral sclerosis.In this paper are presented some works with the aim to integrate brain computer interfaces and mobile robots.The two aim of this project are: (i) to test an improved BCI experience through the help of a physical robot, so that brain signals are stronger stimulate. (ii) to use a remote robot controlled by a highly paralyzed patient via a BCI through a friendly graphic user. Some preliminary experiments are presented in this paper about one of the possible application: a robotic museum guide(PeopleBot and Pioneer3 robot), that can transmit remote visual perceptions to the patient.

This paper presents the localization of a mobile robot while simultaneously mapping the position of the nodes of a Wireless Sensor Network using only range measurements. The robot can estimate the distance to nearby nodes of the Wireless Sensor Network by measuring the Received Signal Strength Indicator (RSSI) of the received radio messages. The RSSI measure is very noisy, especially in an indoor environment due to interference and reflections of the radio signals. We adopted an Extended Kalman Filter SLAM algorithm to integrate RSSI measurements from the different nodes over time, while the robot moves in the environment. A simple pre-processing filter helps in reducing the RSSI variations due to interference and reflections. Successful experiments are reported in which an average localization error less than 1 m is obtained when the SLAM algorithm has no a priori knowledge on the wireless node positions, while a localization error less than 0.5 m can be achieved when the position of the node is initialized close to the their actual position. These results are obtained using a generic path loss model for the transmission channel. Moreover, no internode communication is necessary in the WSN. This can save energy and enables to apply the proposed system also to fully disconnected networks.

Motion blur is a severe problem in images grabbed by legged robots and, in particular, by small humanoid robots. Standard feature extraction and tracking approaches typically fail when applied to sequences of images strongly affected by motion blur. In this paper, we propose a new feature detection and tracking scheme that is robust even to non-uniform motion blur. Furthermore, we developed a framework for visual odometry based on features extracted out of and matched in monocular image sequences. To reliably extract and track the features, we estimate the point spread function (PSF) of the motion blur individually for image patches obtained via a clustering technique and only consider highly distinctive features during matching. We present experiments performed on standard datasets corrupted with motion blur and on images taken by a camera mounted on walking small humanoid robots to show the effectiveness of our approach. The experiments demonstrate that our technique is able to reliably extract and match features and that it is furthermore able to generate a correct visual odometry, even in presence of strong motion blur effects and without the aid of any inertial measurement sensor.

In this article, we propose a new approach to the map building task: the implementation of the Spatial Semantic Hierarchy (SSH), proposed by B. Kuipers, on a real robot fitted with an omnidirectional camera. The original Kuiper's formulation of the SSH was slightly modified, in order to manage in a more efficient way the knowledge the real robot collects while moving in the environment. The sensory data experienced by the robot are transformed by the different levels of the SSH in order to obtain a compact representation of the environment. This knowledge is stored in the form of a topological map and, eventually, of a metrical map. The aim of this article is to show that a catadioptric omnidirectional camera is a good sensor for the SSH and nicely couples with several elements of the SSH. The panoramic view and rotational invariance of our omnidirectional camera makes the identification and labelling of places a simple matter. A deeper insight is that the tracking and identification of events on an omnidirectional image such as occlusions and alignments can be used for the segmentation of continuous sensory image data into the discrete topological and metric elements of a map. The proposed combination of the SSH and omnidirectional vision provides a powerful general framework for robot maping and offers new insights into the concept of “place.” Some preliminary experiments performed with a real robot in an unmodified office environment are presented.

This video shows the interactive robotic sculpture conceived and realized by the artist Albano Guatti. The robotic part was totally developed by people at the IAS-lab and at IT+Robotics according to Guatti's concept. This work is the result of the meeting of robotics and art. The video shows that the robot is able to locate people in the environment, to navigate toward them avoiding the obstacles and to approach them as a polite waiter will do. In fact, the sculpture represents a waiter and a waitress (actually their suits, the statue do not have a body, they represent empty suits). The waiters chat among them (by play different pre-recorded voice files) while wandering in the environment. Once the robot locates a person and get close to her, it asks the customer one out of different pre-recorded questions. The more likely is: Would you like a drink?, that is the title of the sculpture.

This paper considers the properties a multirobot system should exhibit to perform an assigned task cooperatively. Our experiments regard specifically the domain of RoboCup middle-size league (MSL) competitions. But the illustrated techniques can be usefully applied also to other service robotics fields like, for example, videosurveillance. Two issues are addressed in the paper. The former refers to the problem of dynamic role assignment in a team of robots. The latter concerns the problem of sharing the sensory information to cooperatively track moving objects. Both these problems have been extensively investigated over the past years by the MSL robot teams. In our paper, each individual robot has been designed to become reactively aware of the environment configuration. In addition, a dynamic role assignment policy among teammates is activated, based on the knowledge about the best behavior that the team is able to acquire through the shared sensorial information. We present the successful performance of the Artisti Veneti robot team at the MSL Challenge competitions of RoboCup-2003 to show the effectiveness of our proposed hybrid architecture, as well as some tests run in laboratory to validate the omnidirectional distributed vision system which allows us to share the information gathered by the omnidirectional cameras of our robots

The localization problem for an autonomous robot moving in a known environment is a well-studied problem which has seen many elegant solutions. Robot localization in a dynamic environment populated by several moving obstacles, however, is still a challenge for research. In this paper, we use an omnidirectional camera mounted on a mobile robot to perform a sort of scan matching. The omnidirectional vision system finds the distances of the closest color transitions in the environment, mimicking the way laser rangefinders detect the closest obstacles. The similarity of our sensor with classical rangefinders allows the use of practically unmodified Monte Carlo algorithms, with the additional advantage of being able to easily detect occlusions caused by moving obstacles. The proposed system was initially implemented in the RoboCup Middle-Size domain, but the experiments we present in this paper prove it to be valid in a general indoor environment with natural color transitions. We present localization experiments both in the RoboCup environment and in an unmodified office environment. In addition, we assessed the robustness of the system to sensor occlusions caused by other moving robots. The localization system runs in real-time on low-cost hardware.

While probabilistic techniques have previously been investigated extensively for performing inference over the space of metric maps, no corresponding general-purpose methods exist for topological maps. We present the concept of probabilistic topological maps (PTMs), a sample-based representation that approximates the posterior distribution over topologies, given available sensor measurements. We show that the space of topologies is equivalent to the intractably large space of set partitions on the set of available measurements. The combinatorial nature of the problem is overcome by computing an approximate, sample-based representation of the posterior. The PTM is obtained by performing Bayesian inference over the space of all possible topologies, and provides a systematic solution to the problem of perceptual aliasing in the domain of topological mapping. In this paper, we describe a general framework for modeling measurements, and the use of a Markov-chain Monte Carlo algorithm that uses specific instances of these models for odometry and appearance measurements to estimate the posterior distribution. We present experimental results that validate our technique and generate good maps when using odometry and appearance, derived from panoramic images, as sensor measurements.

This paper presents an omnidirectional distributed vision system that learns to navigate a robot in an office-like environment without any knowledge about the calibration of the cameras or the robot control law. The system is composed of several omnidirectional vision agents (implemented with an omnidirectional camera and a computer). The first vision agent learns to control the robot with SARSA(/spl lambda/) reinforcement learning, using the LEM strategy to speed-up learning. Once the first vision agent learnt the correct policy, it transfers its knowledge to the other vision agents. The other vision agents might have different intrinsic and extrinsic camera parameters (that are unknown), so a certain amount of re-learning is needed. Reinforcement learning is well suited for this. In this paper, we present the structure of the learning system and the discussion about the optimal values for the learning parameters. During the experimentation the learning phase of the first agent has been carried out, then the knowledge propagation and the re-learning stage of three different agents have been tested. The experimental results demonstrate the feasibility of the approach and the possibility to port the system on the actual robot and cameras.

One of the most challenging issues in mobile robot navigation is the localization problem in densely populated environments. In this paper, we present a new approach for vision-based localization able to solve this problem. The omnidirectional camera is used as a range finder sensitive to the distance of color transitions, whereas classical range finder;, like lasers or sonars, are sensitive to the distance of the nearest obstacles. The well-known Monte-Carlo localization technique was adapted for this new type of range sensor. The system runs in real time on a low-cost pc. In this paper we present experiments, performed in a crowded RoboCup middle-size field, proving the robustness of the approach to the occlusions of the vision sensor by moving obstacles (e.g other robots); occlusions that are very likely to occur in a real environment. Although, the system was implemented for the RoboCup environment, the system can be used in more general environments.

Reports the steps undertaken in our work aimed to demonstrate the effectiveness of an omnidirectional vision sensor when conjugated with the spatial semantic hierarchy. The spatial semantic hierarchy was proposed by Kuipers (2000) as a method for map building with robots. In our work, a robot builds a topological map of an unknown environment, using the spatial semantic hierarchy and an omnidirectional vision system as the only sensor. We present a new omnidirectional mirror and a new robot. The new mirror was expressly designed for this application, the robot's chassis was designed to create a synergy with the omnidirectional vision sensor. A complete description of our project is reported, underlying the strict link it is possible to create between omnidirectional vision and the spatial semantic hierarchy. Experiments in simulated environments and in real environments produced positive results.