This paper proposes a novel method to refine the 6D pose estimation inferred by an instance-level deep neural network which processes a single RGB image and that has been trained on synthetic images only. The proposed optimization algorithm usefully exploits the depth measurement of a standard RGB-D camera to estimate the dimensions of the considered object, even though the network is trained on a single CAD model of the same object with given dimensions. The improved accuracy in the pose estimation allows a robot to grasp apples of various types and significantly different dimensions successfully; this was not possible using the standard pose estimation algorithm, except for the fruits with dimensions very close to those of the CAD drawing used in the training process. Grasping fresh fruits without damaging each item also demands a suitable grasp force control. A parallel gripper equipped with special force/tactile sensors is thus adopted to achieve safe grasps with the minimum force necessary to lift the fruits without any slippage and any deformation at the same time, with no knowledge of their weight.

The adoption of robots outside their cages in conventional industrial scenarios requires not only safe human-robot interaction but also intuitive human-robot interactive communication. In human-robot collaborative tasks, the objective is to help humans in performing their job with less physical and cognitive effort. A collaborative task can involve the exchange of objects between the robot and the operator. However, the handover operation should be sufficiently intuitive, fluid, and natural for being accepted by the involved humans. Naturalness strongly depends on the speed of the object exchange and the way of communication. For the latter aspect, this paper proposes a multi-modal communication based on visual and haptic cues. Concerning the handover speed requirement, the paper proposes a high-performance visual servoing based on an Extended Kalman Filter (EKF) estimating object speed during the handover and a homography-based object tracking. The object safety is ensured by proper control of the robot grasp force based on a model-based approach exploiting tactile measurements. The same perception modality is also used as a source of haptic cues that make the handover intuitive and natural. Experiments of human-robot handovers through haptic and visual cues communication demonstrate the effectiveness of the proposed approach.

This article investigates the problem of controlling the speed of robots in collaborative workcells for automated manufacturing. The solution is tailored to robotic cells for cooperative assembly of aircraft fuselage panels, where only structural elements are present and robots and humans can share the same workspace, but no physical contact is allowed, unless it happens at zero robot speed. The proposed approach addresses the problem of satisfying the minimal set of requirements of an industrial human–robot collaboration (HRC) task: precision and reliability of human detection and tracking in the shared workspace; correct robot task execution with minimum cycle time while assuring safety for human operators. These requirements are often conflicting with each other. The former does not only concern with safety only but also with the need of avoiding unnecessary robot stops or slowdowns in case of false-positive human detection. The latter, according to the current regulations, concerns with the need of computing the minimum protective separation distance between the human operator and the robots by adjusting their speed when dangerous situations happen. This article proposes a novel fuzzy inference approach to control robot speed enforcing safety while maximizing the level of productivity of the robot minimizing cycle time as well. The approach is supported by a sensor fusion algorithm that merges the images acquired from different depth sensors with those obtained from a thermal camera, by using a machine learning approach. The methodology is experimentally validated in two experiments: the first one at a lab-scale and the second one performed on a full-scale robotic workcell for cooperative assembly of aeronautical structural parts. Note to Practitioners —This article discusses a way to handle human safety specifications versus production requirements in collaborative robotized assembly systems. State-of-the-art (SoA) approaches cover only a few aspects of both human detection and robot speed scaling. The present research work proposes a complete pipeline that starts from a robust human tracking algorithm and scales the robot speed in real time. An innovative multimodal perception system composed of two depth cameras and a thermal camera monitors the collaborative workspace. The speed scaling algorithm is optimized to take on different human behaviors during less risky situations or more dangerous ones to guarantee both operator safety and minimum production time with the aim of better profitability and efficiency for collaborative workstations. The algorithm estimates the operator intention for real-time computation of the minimum protective distance according to the current safety regulations. The robot speed is smoothly changed for the psychological advantages of operators, both in the case of single and multiple workers. The result is a complete system, easily implementable on a standard industrial workcell.

A robust grasp controller for both slipping avoidance and controlled sliding is proposed based on force/tactile feedback only. The model-based algorithm exploits a modified LuGre friction model to consider rotational frictional sliding motions. The modification relies on the Limit Surface concept where a novel computationally efficient method is introduced to compute in real-time the minimum grasping force to balance tangential and torsional loads. The two control modalities are considered by the robot motion planning algorithm that automatically generates robot motions and gripper commands to solve complex manipulation tasks in a material handling application.

This article describes a set of control algorithms for in-hand object handling using a parallel jaw gripper equipped with force/tactile sensors. The control strategy is model based and relies upon the limit surface concept. The LuGre friction model is combined with the limit surface method to set up a dynamic model of soft contact. The model is exploited to estimate the relative velocity of the object with respect to the fingers, so as to control the grip force to counteract possible slipping events due to external disturbances. Force/tactile feedback, the only perception source used by the algorithms, is suitably exploited not only for safe grasping of a variety of objects with uncertain weight and inertial properties, but also for in-hand manipulation actions, like object pivoting or gripper pivoting. Since the algorithm is based on the control of the object velocity, accuracy of the desired object positioning depends on the initial grasp configuration, as well as on the accuracy of the friction model parameters. Such manipulation skills are evaluated in the execution of various pick and place tasks typical of an in-store logistic scenario.

Workspace monitoring is a critical hw/sw component of modern industrial work cells or in service robotics scenarios, where human operators share their workspace with robots. Reliability of human detection is a major requirement not only for safety purposes but also to avoid unnecessary robot stops or slowdowns in case of false positives. The present paper introduces a novel multimodal perception system for human tracking in shared workspaces based on the fusion of depth and thermal images. A machine learning approach is pursued to achieve reliable detection performance in multi-robot collaborative systems. Robust experimental results are finally demonstrated on a real robotic work cell.

The work implements a new real-time flexible motion planning method used for reactive object manipulation in pick and place tasks typical of in-store logistics scenarios such as shelf replenishment of retail stores.This method uses a new hybrid pipeline to recognize and localize an object observed through a depth camera, by integrating and optimizing state of the art techniques.The proposed algorithm guarantees recognition robustness and localization accuracy.The desired object is then manipulated.The motion planner, based on the obstacles detected in the scene, plans a collision-free path towards the target pose.The planned trajectory optimizes a cost function that reflects the best solution among those available and produces natural and smooth path through a smart IK constrained solution which avoids robot unnecessary reconfigurations.A reactive control based on distributed proximity sensors is finally adopted to locally modify the planned trajectory in real time to avoid collisions with uncertain or dynamic obstacles.Experimental results in a supermarket scenario populated with cluttered obstacles demonstrate smoothness of the robot motions and reactive capabilities in a typical fetch and carry task.

Parallel jaw grippers have a limited dexterity, however they can still be used for in-hand manipulation tasks, such as pivoting or other controlled sliding motions of the grasped object. A rotational sliding maneuver is challenging since the grasped object can easily slip if the grip force is not properly adjusted to allow rotational sliding while avoiding translational sliding at the same time. This paper has a twofold aim. First, it intends to refine control algorithms to avoid both rotational and linear slippage, already presented by the authors, by proposing a novel sliding motion model that leads to a grip force as small as possible to avoid slippage, so as to enlarge the set of fragile and deformable objects that can be safely grasped with this approach. Second, the paper exploits the motion model to set up a new algorithm for controlled rotational sliding, thus enabling challenging in-hand manipulation actions. All control algorithms are sensor-based, exploiting a sensorized gripper equipped with a six-axis force/tactile sensor, which provides contact force and torque measurements as well as orientation of the object with respect to the gripper. A set of experiments are executed on a Kuka iiwa showing how the proposed control algorithms are effective to both avoid slippage and allow a controlled sliding motion.