This paper proposes a novel method to refine the 6D pose estimation inferred by an instance-level deep neural network which processes a single RGB image and that has been trained on synthetic images only. The proposed optimization algorithm usefully exploits the depth measurement of a standard RGB-D camera to estimate the dimensions of the considered object, even though the network is trained on a single CAD model of the same object with given dimensions. The improved accuracy in the pose estimation allows a robot to grasp apples of various types and significantly different dimensions successfully; this was not possible using the standard pose estimation algorithm, except for the fruits with dimensions very close to those of the CAD drawing used in the training process. Grasping fresh fruits without damaging each item also demands a suitable grasp force control. A parallel gripper equipped with special force/tactile sensors is thus adopted to achieve safe grasps with the minimum force necessary to lift the fruits without any slippage and any deformation at the same time, with no knowledge of their weight.

Despite modern industries are becoming increasingly automated, wire harness manufacturing processes still rely on manual assembly. One of the reasons behind the difficulty in making the process automatic is that wire harnesses and cable assemblies are highly customized products depending on the application. Hence, realizing an industrial automatic machine for the production of a specific wire harness is not affordable, and manual production remains the most cost-effective. A step towards the automation of wire harness manufacturing is the realization of a system that can be easily adapted to produce different types of cable assemblies. This paper proposes a system composed of a robotic arm, a gripper, and sensorized fingers for executing a wire harness manipulation task. The system can be easily adapted to produce different products by updating the order of operations, the trajectories, and the dimensions/positioning of some low cost mechanical parts.

The adoption of robots outside their cages in conventional industrial scenarios requires not only safe human-robot interaction but also intuitive human-robot interactive communication. In human-robot collaborative tasks, the objective is to help humans in performing their job with less physical and cognitive effort. A collaborative task can involve the exchange of objects between the robot and the operator. However, the handover operation should be sufficiently intuitive, fluid, and natural for being accepted by the involved humans. Naturalness strongly depends on the speed of the object exchange and the way of communication. For the latter aspect, this paper proposes a multi-modal communication based on visual and haptic cues. Concerning the handover speed requirement, the paper proposes a high-performance visual servoing based on an Extended Kalman Filter (EKF) estimating object speed during the handover and a homography-based object tracking. The object safety is ensured by proper control of the robot grasp force based on a model-based approach exploiting tactile measurements. The same perception modality is also used as a source of haptic cues that make the handover intuitive and natural. Experiments of human-robot handovers through haptic and visual cues communication demonstrate the effectiveness of the proposed approach.

The increasing interest in autonomous robots with a high number of degrees of freedom for industrial applications and service robotics demands control algorithms to handle multiple tasks as well as hard constraints efficiently. This article presents a general framework in which both kinematic (velocity- or acceleration-based) and dynamic (torque-based) control of redundant robots are handled in a unified fashion. The framework allows for the specification of redundancy resolution problems featuring a hierarchy of arbitrary (equality and inequality) constraints, arbitrary weighting of the control effort in the cost function and an additional input used to optimize possibly remaining redundancy. To solve such problems, a generalization of the saturation in the null space algorithm is introduced, which extends the original method according to the features required by our general control framework. Variants of the developed algorithm are presented, which ensure both efficient computation and optimality of the solution. Experiments on a KUKA LBRiiwa robotic arm, as well as simulations with a highly redundant mobile manipulator are reported.

This article investigates the problem of controlling the speed of robots in collaborative workcells for automated manufacturing. The solution is tailored to robotic cells for cooperative assembly of aircraft fuselage panels, where only structural elements are present and robots and humans can share the same workspace, but no physical contact is allowed, unless it happens at zero robot speed. The proposed approach addresses the problem of satisfying the minimal set of requirements of an industrial human–robot collaboration (HRC) task: precision and reliability of human detection and tracking in the shared workspace; correct robot task execution with minimum cycle time while assuring safety for human operators. These requirements are often conflicting with each other. The former does not only concern with safety only but also with the need of avoiding unnecessary robot stops or slowdowns in case of false-positive human detection. The latter, according to the current regulations, concerns with the need of computing the minimum protective separation distance between the human operator and the robots by adjusting their speed when dangerous situations happen. This article proposes a novel fuzzy inference approach to control robot speed enforcing safety while maximizing the level of productivity of the robot minimizing cycle time as well. The approach is supported by a sensor fusion algorithm that merges the images acquired from different depth sensors with those obtained from a thermal camera, by using a machine learning approach. The methodology is experimentally validated in two experiments: the first one at a lab-scale and the second one performed on a full-scale robotic workcell for cooperative assembly of aeronautical structural parts. Note to Practitioners —This article discusses a way to handle human safety specifications versus production requirements in collaborative robotized assembly systems. State-of-the-art (SoA) approaches cover only a few aspects of both human detection and robot speed scaling. The present research work proposes a complete pipeline that starts from a robust human tracking algorithm and scales the robot speed in real time. An innovative multimodal perception system composed of two depth cameras and a thermal camera monitors the collaborative workspace. The speed scaling algorithm is optimized to take on different human behaviors during less risky situations or more dangerous ones to guarantee both operator safety and minimum production time with the aim of better profitability and efficiency for collaborative workstations. The algorithm estimates the operator intention for real-time computation of the minimum protective distance according to the current safety regulations. The robot speed is smoothly changed for the psychological advantages of operators, both in the case of single and multiple workers. The result is a complete system, easily implementable on a standard industrial workcell.

A robust grasp controller for both slipping avoidance and controlled sliding is proposed based on force/tactile feedback only. The model-based algorithm exploits a modified LuGre friction model to consider rotational frictional sliding motions. The modification relies on the Limit Surface concept where a novel computationally efficient method is introduced to compute in real-time the minimum grasping force to balance tangential and torsional loads. The two control modalities are considered by the robot motion planning algorithm that automatically generates robot motions and gripper commands to solve complex manipulation tasks in a material handling application.

Driving a car is a complex and potentially risky activity in people's everyday life, and it requires the full involvement of physiological and cognitive resources. Any loss of these resources can cause traffic accidents. For example, drowsy driving affects the ability to adapt, predict and react to unexpected events. A solution to this problem is the adoption of Advanced Driver Assistance Systems (ADAS), which can warn the driver if sleepiness is detected. Thus, they should include a Driver Monitoring System (DMS) to understand, measure and monitor human behaviour in different scenarios. This article is focused on detecting driver drowsiness by using non-intrusive measures such as the behavioural approach, as it is the most promising solution to use in real vehicles. The developed framework allows the extraction of drowsiness-related measures by analysing the driver's face with a standard camera. First, a face detection stage identifies the driver face in a video frame. Then, a set of facial landmarks locations are identified. These landmark points are used to estimate the head orientation and to detect when a blink occurs. By monitoring properly defined ocular variables, the degree of driver drowsiness is detected through a Fuzzy Inference System (FIS).

This article describes a set of control algorithms for in-hand object handling using a parallel jaw gripper equipped with force/tactile sensors. The control strategy is model based and relies upon the limit surface concept. The LuGre friction model is combined with the limit surface method to set up a dynamic model of soft contact. The model is exploited to estimate the relative velocity of the object with respect to the fingers, so as to control the grip force to counteract possible slipping events due to external disturbances. Force/tactile feedback, the only perception source used by the algorithms, is suitably exploited not only for safe grasping of a variety of objects with uncertain weight and inertial properties, but also for in-hand manipulation actions, like object pivoting or gripper pivoting. Since the algorithm is based on the control of the object velocity, accuracy of the desired object positioning depends on the initial grasp configuration, as well as on the accuracy of the friction model parameters. Such manipulation skills are evaluated in the execution of various pick and place tasks typical of an in-store logistic scenario.

Workspace monitoring is a critical hw/sw component of modern industrial work cells or in service robotics scenarios, where human operators share their workspace with robots. Reliability of human detection is a major requirement not only for safety purposes but also to avoid unnecessary robot stops or slowdowns in case of false positives. The present paper introduces a novel multimodal perception system for human tracking in shared workspaces based on the fusion of depth and thermal images. A machine learning approach is pursued to achieve reliable detection performance in multi-robot collaborative systems. Robust experimental results are finally demonstrated on a real robotic work cell.

In this paper, we introduce the problem of cross-modal visuo-tactile object recognition with robotic active exploration. With this term, we mean that the robot observes a set of objects with visual perception, and later on, it is able to recognize such objects only with tactile exploration, without having touched any object before. Using a machine learning terminology, in our application, we have a visual training set and a tactile test set, or vice versa. To tackle this problem, we propose an approach constituted by four steps: finding a visuo-tactile common representation, defining a suitable set of features, transferring the features across the domains, and classifying the objects. We show the results of our approach using a set of 15 objects, collecting 40 visual examples and five tactile examples for each object. The proposed approach achieves an accuracy of 94.7%, which is comparable with the accuracy of the monomodal case, i.e., when using visual data both as training set and test set. Moreover, it performs well compared to the human ability, which we have roughly estimated carrying out an experiment with ten participants.

The work implements a new real-time flexible motion planning method used for reactive object manipulation in pick and place tasks typical of in-store logistics scenarios such as shelf replenishment of retail stores.This method uses a new hybrid pipeline to recognize and localize an object observed through a depth camera, by integrating and optimizing state of the art techniques.The proposed algorithm guarantees recognition robustness and localization accuracy.The desired object is then manipulated.The motion planner, based on the obstacles detected in the scene, plans a collision-free path towards the target pose.The planned trajectory optimizes a cost function that reflects the best solution among those available and produces natural and smooth path through a smart IK constrained solution which avoids robot unnecessary reconfigurations.A reactive control based on distributed proximity sensors is finally adopted to locally modify the planned trajectory in real time to avoid collisions with uncertain or dynamic obstacles.Experimental results in a supermarket scenario populated with cluttered obstacles demonstrate smoothness of the robot motions and reactive capabilities in a typical fetch and carry task.

Parallel jaw grippers have a limited dexterity, however they can still be used for in-hand manipulation tasks, such as pivoting or other controlled sliding motions of the grasped object. A rotational sliding maneuver is challenging since the grasped object can easily slip if the grip force is not properly adjusted to allow rotational sliding while avoiding translational sliding at the same time. This paper has a twofold aim. First, it intends to refine control algorithms to avoid both rotational and linear slippage, already presented by the authors, by proposing a novel sliding motion model that leads to a grip force as small as possible to avoid slippage, so as to enlarge the set of fragile and deformable objects that can be safely grasped with this approach. Second, the paper exploits the motion model to set up a new algorithm for controlled rotational sliding, thus enabling challenging in-hand manipulation actions. All control algorithms are sensor-based, exploiting a sensorized gripper equipped with a six-axis force/tactile sensor, which provides contact force and torque measurements as well as orientation of the object with respect to the gripper. A set of experiments are executed on a Kuka iiwa showing how the proposed control algorithms are effective to both avoid slippage and allow a controlled sliding motion.

In-hand manipulation is certainly one of the most challenging problems in robotic manipulation. Solutions to this problem depend on the specific device used to grab the object, but nowadays, the trend is to exploit not only the gripper but also external constraints, such as other objects in the environment or external forces, like gravity. This allows a robot to manipulate an object even with very simple grippers, like a parallel gripper. Nevertheless, even for a simple grasping task, which aims at grabbing the object with a given fixed orientation or for executing a controlled slip, information on the contact between the fingers of the gripper and the object is relevant. In these cases, both linear and rotational slipping should be controlled during the grasping phase and during the motion phase. The present paper proposes a control strategy for the first objective, namely slipping avoidance. The strategy is based on contact information provided by a six-axis force/tactile sensor, able to measure contact force and torque as well as able to provide information on the contact geometry, that means orientation of the object with respect to the gripper. Experiments on a parallel gripper sensorized with a new force/tactile sensor and mounted on a Kuka iiwa show how the strategy successfully allows the robot to safely manipulate a rigid object in various friction conditions of its surface.

In this work, we propose a framework to deal with cross-modal visuo-tactile object recognition. By cross-modal visuo-tactile object recognition, we mean that the object recognition algorithm is trained only with visual data and is able to recognize objects leveraging only tactile perception. The proposed cross-modal framework is constituted by three main elements. The first is a unified representation of visual and tactile data, which is suitable for cross-modal perception. The second is a set of features able to encode the chosen representation for classification applications. The third is a supervised learning algorithm, which takes advantage of the chosen descriptor. In order to show the results of our approach, we performed experiments with 15 objects common in domestic and industrial environments. Moreover, we compare the performance of the proposed framework with the performance of 10 humans in a simple cross-modal recognition task.

This paper presents a novel approach for safeguarding a mobile manipulator. To overcome the shortcomings of state-of-the-art safeguarding techniques, we present a dynamic approach capable of covering different situations and adapting itself to the situation at hand. The proposed approach is based on the continuous dynamic update of a safety field using both exteroceptive and proprioceptive data. The proprioceptive data are the platform velocity vector and the manipulator position; the exteroceptive data refers to the environment in which the mobile manipulator is moving, namely distances to obstacles acquired by laser scanners. The effectiveness of the proposed approach has been validated in an industrial test case scenario in which the robot had to execute a fetch and carry task while safely coexisting with human workers. Experimental results proved that the use of dynamic protection fields allowed the robot to carry out its tasks with a significant reduction (about 50%) of the execution time.

The main objective of this paper is to propose an experimental modal analysis procedure, based on the use of a gray-box model for flexible structures. The described approach presents interesting advantages with respect to commercial solutions: ease of use due to the low number of parameters to set for an identification session; no need for expert users, even in the presence of particular cases such as double modes, since it does not use a stabilization diagram to be elaborated; use of a gray-box model whose unknown parameters have a clear physical meaning. All these characteristics are discussed in the paper, and the performance of the proposed procedure has been evaluated by using experimental data available from a non-trivial standard benchmark. The results have been compared with those obtained by using a commercial tool.

This paper proposes an experimental study of slipping avoidance algorithms based on force/tactile perception data. The claim is that contact force measurements alone or tactile data alone are not sufficient for an effective slipping avoidance strategy in real world conditions. Integrated force/tactile sensors able to provide measurements of both the contact force vector and spatially distributed tactile maps are the key enabling technology for efficient slipping avoidance control algorithms that can actually work with real world objects under no restricting assumption on the contact geometry or with unknown physical properties of the objects. The paper proposes a new slipping avoidance control scheme, which usefully exploits an integrated force/tactile sensor mounted on the parallel gripper of a Kuka youBot. The results show how the strategy successfully allows the robot to safely manipulate real-world objects, both rigid and compliant, in various friction conditions of their surface, both stable and slippery.

The stability of discrete time kinematic sensor-based control of robots is investigated in this paper. A hierarchical inner-loop/outer-loop control architecture common for a generic robotic system is considered. The inner loop is composed of a servo-level joint controller and higher level kinematic feedback is performed in the outer loop. Stability results derived in this paper are of interest in several applications including visual servoing problems, redundancy control, and coordination/synchronization problems. The stability of the overall system is investigated taking into account input/output delays and the inner loop dynamics. A necessary and sufficient condition that the gain of the outer feedback loop has to satisfy to ensure local stability is derived. Experiments on a Kuka K-R16 manipulator have been performed in order to validate the theoretical findings on a real robotic system and show their practical relevance.

The paper presents a novel slipping control algorithm based on the exploitation of the tactile sensor integrated into the DEXMART anthropomorphic robotic hand. The Extended Kalman Filter (EKF) is used to solve in real-time the nonlinear model of the sensor, allowing to estimate the contact geometry variables and the friction coefficient. The innovative proposed slipping control is based on the use of the estimation error of the EKF as an indicator of incipient slipping events. The control algorithm computes the suitable grip force based on the estimation error. The effectiveness of the proposed approach is shown with experimental results.

The aim of this paper is to investigate the discrete-time stability of robot motion control in the task space. The control system has been modeled as a classical inner-loop/outer-loop architecture, adopted in several industrial robotic systems. The inner-loop is composed of a servo-level joint controller, and higher level kinematic feedback is performed in the outer-loop. Heterogeneous dynamics is considered in the inner-loop, which can for instance describe redundant coordination/synchronization control systems with cooperative robots with non-identical dynamical responses. There are surprisingly few discrete-time stability results in the current state-of-the-art for this popular control architecture. The qualitative effects of the inner-loop dynamics on the overall stability of the system is investigated, and improved outer-loop feedback gain margins are derived.

The purpose of this paper is to provide a convergence analysis of classical inverse kinematics algorithms for redundant robots, whose stability is usually proved only in the continuous-time domain, thus neglecting limits of the actual implementation in the discrete time, whereas the convergence analysis carried out in this paper in the discrete-time domain provides a method to find bounds on the gain of the closed-loop inverse kinematics algorithms in relation to the sampling time. It also provides an estimation of the region of attraction (without resorting to Lyapunov arguments), i.e., upper bounds on the initial task space error. Simulations on an 11-degree-of-freedom manipulator are performed to show how the found bounds on the gain are not too restrictive.

The purpose of this paper is to present preliminary results on the use of a torque sensor based on a Bragg grating for torque control applications of tendon-driven mechanisms. Owing to the minimally invasive nature of optical fibres, one of the most promising applications can be the integration of the sensor into anthropomorphic robotic hands for accurate impedance or compliance control. In fact, the sensing element of the proposed torque sensor can be easily bonded directly to the tendon following its natural routing with a significantly reduced invasiveness with respect to conventional sensors. These are usually based on strain gauges, which are cumbersome and require additional mechanical components and interfaces in order to provide the necessary measurement, while the Bragg sensor is embedded into the optical fibre whose typical diameter is about 125 mum which allows its integration into the tendon itself. The experimental results presented here have been obtained on a simple test-bench realized by using off-the-shelf and cheap components conceived to demonstrate the potentiality of the sensor and its effectiveness in an actual compliance control scheme.

The goal of the paper is the application of a geometrically consistent impedance concept to control interaction with the environment of a rigid object manipulated by a dual-robot system. A six-DOF impedance is specified at the object level to confer a compliant behavior for both the translational and the rotational motion when an external force and moment occurs at the contact. Geometric consistency is ensured thanks to the use of the unit quaternion to describe object frame orientation. The resulting object motion is decomposed into the equivalent motions at the end effectors of the two robots, via a task-oriented formulation. The control scheme is derived according to an inverse dynamics strategy with adoption of an inner motion loop providing robustness to unmodeled dynamics and disturbances. Experimental results on the two industrial robots available in the lab are discussed.

Recent research in industrial robotics aims at the involvement of additional sensory devices to improve robustness, flexibility and performance of common robot applications. Many different sensors have been developed over the past years to fit the requirements of different but very specific tasks. Special seam tracking sensors support the robot in welding applications. Vision systems are common in quality control and inspection. Force/torque sensors mounted to a robot's wrist are still an exception and limited to the fields of scientific research. Compared to the number of annual robot sales the number of sensor equipped robots is still negligible, although the benefits of sensory feedback are obvious. In this paper we introduce a general approach to tackle the problem of sensor-based robot assembly. We realized a flexible assembly cell that includes a variety of different sensors for mating with moving parts.

Focuses on the problem of tracking a desired end-effector position and orientation for a robot manipulator. A nonminimal parameterization of the orientation-the unit quaternion-is used to design the control law. Tracking control typically requires full-state measurements, i.e., end-effector position and orientation as well as linear and angular velocities. However, while position and orientation measurements are available, often velocities have to be estimated. In the paper, an observer-controller scheme is proposed which ensures tracking in the task space without using velocity measurements. The performance of the proposed scheme is evaluated both in simulation and through experimental tests on an industrial robot.

This work is focused on impedance control of redundant manipulators. A spatial impedance formulation is presented where general 6-DOF end-effector tasks can be handled. A singularity-free angle/axis representation of end-effector orientation is used which allows geometric task consistency to be preserved. An inverse dynamics control with a dynamically consistent inverse of the geometric Jacobian matrix is developed with the adoption of an inner loop acting on the end-effector position and orientation conferring robustness to unmodeled dynamics and external disturbances. Stabilization of null-space velocities is ensured and utilization of redundant degrees of mobility is carried out to optimize an additional task function. Experimental results on a seven-joint industrial robot with force/torque sensor are discussed.

A new approach to 6-DOF impedance control is proposed, where the end-effector orientation displacement is derived from the rotation matrix expressing the mutual orientation between the compliant frame and the desired frame. An alternative Euler angles-based description is proposed which mitigates the effects of representation singularities. Then, a class of angle/axis representations are considered to derive the dynamic equation for the rotational part of a 6-DOF impedance at the end effector, using an energy-based argument. The unit quaternion representation is selected to further analyze the properties of the rotational impedance. The resulting impedance controllers are designed according to an inverse dynamics strategy with contact force and moment measurements, where an inner loop acting on the end-effector position and orientation error is adopted to confer robustness to unmodeled dynamics and external disturbances. Experiments on an industrial robot were carried out, and the results of case studies are discussed.

The work is aimed at studying the problem of controlling the moment and the orientation of a robot manipulator whose end effector is in contact with a compliant environment. Differently from classical operational space formulations, a geometrical approach is pursued where orientation displacements are described in terms of unit quaternions. Regulation to a constant desired moment and tracking of a time-varying desired orientation trajectory is achieved, and the convergence of the closed-loop system is analyzed. Simulation results for a six-joint industrial robot are developed and an experimental test is carried out to demonstrate the effectiveness of the proposed approach in a practical contact task.