This paper introduces the Proactive Assistance through action-Completion Estimation (PACE) framework, designed to enhance human-robot collaboration through real-time monitoring of human progress. PACE incorporates a novel method that combines Dynamic Time Warping (DTW) with correlation analysis to track human task progression from hand movements. PACE trains a reinforcement learning policy from limited demonstrations to generate a proactive assistance policy that synchronizes robotic actions with human activities, minimizing idle time and enhancing collaboration efficiency. We validate the framework through user studies involving 12 participants, showing significant improvements in interaction fluency, reduced waiting times, and positive user feedback compared to traditional methods.

Despite the importance of human-robot collaborative transportation of flexible material in many industrial scenarios, many works in the literature assume a passive role for the robot during the collaboration. The robot can only follow the human partner, without providing assistance in the more challenging phase of the collaboration such as precise material positioning. This work presents a framework for co-transportation, proposing a distance-based policy for dynamic leader role allocation through the task. For large distances from the target pose, the robot is mainly controlled by vision-based manual guidance exploiting haptic feedback and 3D human pose information; instead, close to the target material position, the robot acts as a leader guiding the human operator. The proposed framework is evaluated considering a carbon fiber draping task, which requires both co-transportation and precise positioning of flexible materials. Experimental results demonstrate how the robot leading the task in the final stage allows to achieve high task efficiency and alleviates human stress in the execution of the task.

This paper presents MEMROC (Multi-Eye to Mobile RObot Calibration), a novel motion-based calibration method that simplifies the process of accurately calibrating multiple cameras relative to a mobile robot’s reference frame. MEMROC utilizes a known calibration pattern to facilitate accurate calibration with a lower number of images during the optimization process. Additionally, it leverages robust ground plane detection for comprehensive 6-DoF extrinsic calibration, overcoming a critical limitation of many existing methods that struggle to estimate the complete camera pose. The proposed method addresses the need for frequent recalibration in dynamic environments, where cameras may shift slightly or alter their positions due to daily usage, operational adjustments, or vibrations from mobile robot movements. MEMROC exhibits remarkable robustness to noisy odometry data, requiring minimal calibration input data. This combination makes it highly suitable for daily operations involving mobile robots. A comprehensive set of experiments on both synthetic and real data proves MEMROC’s efficiency, surpassing existing state-of-the-art methods in terms of accuracy, robustness, and ease of use. To facilitate further research, we have made our code publicly available1.

Robotic waste sorting poses significant challenges in both perception and manipulation, given the extreme variability of objects that should be recognized on a cluttered conveyor belt. While deep learning has proven effective in solving complex tasks, the necessity for extensive data collection and labeling limits its applicability in real-world scenarios like waste sorting. To tackle this issue, we introduce a data augmentation method based on a novel GAN architecture called wasteGAN. The proposed method allows to increase the performance of semantic segmentation models, starting from a very limited bunch of labeled examples, such as few as 100. The key innovations of wasteGAN include a novel loss function, a novel activation function, and a larger generator block. Overall, such innovations helps the network to learn from limited number of examples and synthesize data that better mirrors real-world distributions. We then leverage the higher-quality segmentation masks predicted from models trained on the wasteGAN synthetic data to compute semantic-aware grasp poses, enabling a robotic arm to effectively recognizing contaminants and separating waste in a real-world scenario. Through comprehensive evaluation encompassing dataset-based assessments and real-world experiments, our methodology demonstrated promising potential for robotic waste sorting, yielding performance gains of up to 5.8% in picking contaminants. The project page is available at https://github.com/bach05/wasteGAN.git.

This paper proposes a task planning framework for collaborative Human-Robot scenarios, specifically focused on assembling complex systems such as furniture. The human is characterized as an uncontrollable agent, implying for example that the agent is not bound by a pre-established sequence of actions and instead acts according to its own preferences. Meanwhile, the task planner computes reactively the optimal actions for the collaborative robot to efficiently complete the entire assembly task in the least time possible.We formalize the problem as a Discrete Event Markov Decision Problem (DE-MDP), a comprehensive framework that incorporates a variety of asynchronous behaviors, human change of mind, and failure recovery as stochastic events. Although the problem could theoretically be addressed by constructing a graph of all possible actions, such an approach would be constrained by computational limitations. The proposed formulation offers an alternative solution utilizing Reinforcement Learning to derive an optimal policy for the robot. Experiments were conducted both in simulation and on a real system with human subjects assembling a chair in collaboration with a 7-DoF manipulator.

This paper presents an accurate and precise hand-eye calibration technique based on minimization of the reprojection error. Unlike traditional hand-eye calibration, the proposed method does not require an explicit estimate of the camera pose for each input image because it does not rely on mathematical description and problem formulation commonly used in standard hand-eye calibration algorithms. The proposed method is based on a nonlinear optimization problem, so that the estimation problem can be solved efficiently and robustly, and can be easily extended to different camera-robot setups (e.g., eye-on-base or eye-in-hand). An extensive evaluation based on simulated and real experiments has been performed, proving its good estimation accuracy in terms of reprojection error. The experimental results with real robots show that the proposed method is applicable to relevant industrial contexts and improves the quality and precision of the camera-robot transformation estimation with respect to state-of-the-art approaches.

In this work we present an industrial system for the inspection of composite parts in the aerospace industry, based on X-ray sensors and robotic manipulators. Such system is designed to identify any type of defects such as, missing gluing, core cell deformation, cracks or foreign objects, which may occur between layers of which these objects are composed. The inspection process involves back-projection of X-ray images onto the 3D CAD model of the inspected part, to directly locate the defects on the part itself. The complete system has been implemented in a real industrial workcell that involves two synchronized robots equipped with a X-ray source-detector system. The two robots move autonomously along a pre-computed trajectory without any human intervention, and the back-projection of the acquired images is efficiently performed at run-time using the proposed algorithm. The experiments demonstrate that the X-ray images back-projection is successful and can effectively replace standard manually guided inspections. This has a high impact on the factory automation cycle since it helps to reduce the effort and time needed for each inspection task. This work is part of a EU funded project called SPIRIT.

Semantic segmentation is a problem which is getting more and more attention in the computer vision community. Nowadays, deep learning methods represent the state of the art to solve this problem, and the trend is to use deeper networks to get higher performance. The drawback with such models is a higher computational cost, which makes it difficult to integrate them on mobile robot platforms. In this work we want to explore how to obtain lighter deep learning models without compromising performance. To do so we will consider the features used in the 3D Entangled Forests algorithm and we will study the best strategies to integrate these within FuseNet deep network. Such new features allow us to shrink the network size without loosing performance, obtaining hence a lighter model which achieves state-of-the-art performance on the semantic segmentation task and represents an interesting alternative for mobile robotics applications, where computational power and energy are limited.

Object detection plays a crucial role in the development of Electronic Travel Aids (ETAs), capable to guide a person with visual impairments towards a target object in an unknown indoor environment. In such a scenario, the object detector runs on a mobile device (e.g. smartphone) and needs to be fast, accurate, and, most importantly, lightweight. Nowadays, Deep Neural Networks (DNN) have become the state-of-the-art solution for object detection tasks, with many works improving speed and accuracy by proposing new architectures or extending existing ones. A common strategy is to use deeper networks to get higher performance, but that leads to a higher computational cost which makes it impractical to integrate them on mobile devices with limited computational power. In this work we compare different object detectors to find a suitable candidate to be implemented on ETAs, focusing on lightweight models capable of working in real-time on mobile devices with a good accuracy. In particular, we select two models: SSD Lite with Mobilenet V2 and Tiny-DSOD. Both models have been tested on the popular OpenImage dataset and a new dataset, named L-CAS Office dataset, collected to further test models' performance and robustness in a real scenario inspired by the actual perception challenges of a user with visual impairments.

This article presents an integrated vision-based guiding system for aerial manipulation. More specifically, a 4 DoF planar dexterous manipulator, with a stereo camera attached on the end-effector, is endowed to a multirotor aerial platform enabling active manipulation capabilities. The proposed novel approach combines a visual processing scheme for object detection and tracking, as well as a manipulator positioning for allowing the aerial platform to approach the surface of interaction efficiently. In the developed scheme, the object detection is based on correlation filters to track the target robustly, while the depth information, from the stereo camera on board the manipulator, is used to extract the centroid of the manipulated object, compute its relative configuration with respect to the UAV and align the end-effector properly with the grasping point. The effectiveness of the proposed scheme is demonstrated in multiple experimental trials and simulations, highlighting it's applicability towards autonomous aerial manipulation.