Fine dexterous manipulation requires reactive control based on rich sensing of manipulator-object interactions. Tactile sensing arrays provide rich contact information across the manipulator’s surface. However their implementation faces two main challenges: accurate force estimation across complex surfaces like robotic hands, and integration of these estimates into reactive control loops. We present a data-efficient calibration method that enables rapid, full-array force estimation across varying geometries, providing online feedback that accounts for non-linearities and deformation effects. Our force estimation model serves as feedback in an online closed-loop control system for interaction force tracking. The accuracy of our estimates is independently validated against measurements from a calibrated force-torque sensor. Using the Allegro Hand equipped with Xela uSkin sensors, we demonstrate precise force application through an admittance control loop running at 100Hz, achieving up to 0.12±0.08 [N] error margin—results that show promising potential for dexterous manipulation.

In agricultural robotics, effective observation and localization of fruits present challenges due to occlusions caused by other parts of the tree, such as branches and leaves. These occlusions can result in false fruit localization or impede the robot from picking the fruit. The objective of this work is to push away branches that block the fruit’s view to increase their visibility. Our setup consists of an RGB-D camera and a robot arm. First, we detect the occluded fruit in the RGB image and estimate its occluded part via a deep learning generative model in the depth space. The direction to push to clear the occlusions is determined using classic image processing techniques. We then introduce a 3D extension of the 2D Hough transform to detect straight line segments in the point cloud. This extension helps detect tree branches and identify the one mainly responsible for the occlusion. Finally, we clear the occlusion by pushing the branch with the robot arm. Our method uses a combination of deep learning for fruit appearance estimation, classic image processing for push direction determination, and 3D Hough transform for branch detection. We validate our perception methods through real data under different lighting conditions and various types of fruits (i.e. apple, lemon, orange), achieving improved visibility and successful occlusion clearance. We demonstrate the practical application of our approach through a real robot branch pushing demonstration.

Literature highlights how virtual and augmented settings offer engaging solutions to improve one's feeling of an artificial limb embodiment. In this paper, we explored the potential of a setting for Spatial Augmented Reality (SAR, where a display augments a surface without making the user wear any visor) in two conditions of a lower limb ownership training involving subjects without disabilities. In the first condition, the subject must contract the quadriceps of a leg for commanding (through electromyography, EMG) a virtual leg (a 3D model of the Hybrid Knee prosthesis) to kick a virtual wall: each collision corresponds to a vibratory feedback on the thigh (a position defined for upcoming tests with transfemural amputees). The second condition adds a social context to engage the user: the subject is asked to cooperate with another (fictional) player to kick on the same virtual wall. Subjective (through questionnaires) and objective (according to the number of kicks as a performance index, and the proprioceptive drift as an embodiment index) assessments have been performed before a rubber leg illusion test. Overall, we observed how the cooperative task can engage the subject to be more active. However, this condition can reduce the impact of the training on the embodiment itself, probably because the social task generates a distraction. Nevertheless, such findings suggest the possibility to alternate these two tasks in the same session to increase the duration of a prosthetic embodiment training.

Can robots mold soft plastic materials by shaping depth images? The short answer is no: current day robots can not. In this article, we address the problem of shaping plastic material with an anthropomorphic arm/hand robot, which observes the material with a fixed depth camera. Robots capable of molding could assist humans in many tasks, such as cooking, scooping, or gardening. Yet, the problem is complex, due to its high-dimensionality at both perception and control levels. To address it, we design three alternative data-based methods for predicting the effect of robot actions on the material. Then, the robot can plan the sequence of actions and their positions, to mold the material into a desired shape. To make the prediction problem tractable, we rely on two original ideas. First, we prove that under reasonable assumptions, the shaping problem can be mapped from point cloud to depth image space, with many benefits (simpler processing, no need for registration, lower computation time, and memory requirements). Second, we design a novel, simple metric for quickly measuring the distance between two depth images. The metric is based on the inherent point cloud representation of depth images, which enables direct and consistent comparison of image pairs through a nonuniform scaling approach, and therefore opens promising perspectives for designingdepth image-basedrobot controllers. We assess our approach in a series of unprecedented experiments, where a robotic arm/hand molds flour from initial to final shapes, either with its own dataset, or by transfer learning from a human dataset. We conclude the article by discussing the limitations of our framework and those of current day hardware, which make human-like robot molding a challenging open research problem.

In this article, we introduce a new sensor-based control method that regulates (by means of robot motion) the temperature of objects that are subject to a radiative heat source. This valuable sensorimotor capability is needed in many industrial, dermatology, and field robot applications, and it is an essential component for creating machines with advanced thermomotor intelligence. To this end, we derive a geometric-thermal-motor model, which describes the relation between the robot’s active configuration and the produced dynamic thermal response. We then use the model to guide the design of two new thermal servoing controllers (one model-based and one adaptive), and analyze their stability with Lyapunov theory. To validate our method, we report a detailed experimental study with a robotic manipulator conducting autonomous thermal servoing tasks. We show that the temperature of multiple objects with unknown thermophysical properties attached to the same end-effector can be effectively regulated by controlled robot motion. Although thermal sensing is a mature technology in many industrial thermal engineering applications, its use as a feedback signal for robot control has not been sufficiently studied in the literature. To the best of our knowledge, this is the first time that temperature regulation is formulated as a motion control problem for robots.

Human physical assistance requires the assistant to tune both his trajectory and impedance in order to assist an individual as well as be guided by him. In this study we propose a controller for teleoperated human assistance that allows the assistant to guide the assisting robot in both trajectory and impedance. We propose to use the inherent perturbations in the task, induced by the elderly or stroke patient, for impedance estimation, while a simple neuroscience based filter allows the reference estimation of the operator. We tested our impedance estimation and the controller as a whole in two experiments in which a human operator guided a robot suffering force perturbations that simulated a human patient.

This paper addresses the recognition of daily-life objects by a robot equipped with tactile sensors. The main contribution is a deep learning framework that can recognize objects already touched as well as objects never touched before. To this end, we train a deconvolutional neural network that generates synthetic tactile data for novel classes. Then, we use both these synthetic data and the real data collected by touching objects, to train a convolutional neural network to recognize both known (trained) objects and novel objects. Furthermore, we propose a method for integrating newly encountered data into novel classes. Finally, we evaluate the framework using the largest available dataset of tactile objects descriptions.

Information technologies are permeating all aspects of manufacturing systems as well as other fields, expediting the generation of industrial big data. Traditionally, the devices collected the sensor data from various sources, and information fusion was then performed. This incurs higher burden of time and storage cost. Recently, more and more intelligent devices are equipped in the industrial environment. This provides more opportunities for better data collection and processing for industrial intelligence. Active perception technology, which performs control strategies on the data acquisition process, enables the devices to seamlessly integrate the perception and action to reach high-level goals rather than to accomplish low-level commands. It helps to select more useful information and may save the life of the sensors. However, there exist many unsolved challenging problems since the feedback is performed on complex processed sensory data, i.e., various extracted features.

This paper contributes to the field of physical human-robot collaboration. We present a complete control framework, which aims at making humanoid robots capable of carrying objects together with humans. First, we design a template identifying the primitive subtasks necessary for collaborative carrying. Then, these subtasks are formulated as constrained optimization problems for controlling the whole-body motion of a humanoid robot. The subtasks include two walking pattern generators that account for physical collaboration, as well as posture and grasping controllers. Finally, we validate our framework in a variety of collaborative carrying experiments, using the HRP-4 humanoid robot.

This study proposes a visuo-tactile Zero-Shot object recognition framework. The proposed framework recognizes a set of novel objects for which no tactile or visual training data are available. It uses visuo-tactile training data collected from known objects to recognize the novel ones, given their attributes. This framework extends the haptic Zero-Shot Learning framework that we proposed in [1] with vision, which enables a multimodal recognition system. In our test with the PHAC-2 dataset, the system was able to get a recognition accuracy of 72% among 6 objects that were never touched or seen during the training phase.

This paper represents a step towards vision-based manipulation of plastic materials. Manipulating deformable objects is made challenging by: 1) the absence of a model for the object deformation, 2) the inherent difficulty of visual tracking of deformable objects, 3) the difficulty in defining a visual error and 4) the difficulty in generating control inputs to minimise the visual error. We propose a novel representation of the task of manipulating deformable objects. In this preliminary case study, the shaping of kinetic sand, we assume a finite set of actions: pushing, tapping and incising. We consider that these action types affect only a subset of the state, i.e., their effect does not affect the entire state of the system (specialized actions). We report the results of a user study to validate these hypotheses and release the recorded dataset. The actions (pushing, tapping and incising) are clearly adopted during the task, although it is clear that 1) participants use also mixed actions and 2) actions' effects can marginally affect the entire state, requesting a relaxation of our specialized actions hypothesis. Moreover, we compute task errors and corresponding control inputs (in the image space) using image processing. Finally, we show how machine learning can be applied to infer the mapping from error to action on the data extracted from the user study.

For successful physical human-robot interaction, the capability of a robot to understand its environment is imperative. More importantly, the robot should extract from the human operator as much information as possible. A reliable 3D skeleton extraction is essential for a robot to predict the intentions of the operator while s/he moves toward the robot or performs a meaningful gesture. For this purpose, we have integrated a time-of-flight depth camera with a state-of-the-art 2D skeleton extraction library namely Openpose, to obtain 3D skeletal joint coordinates reliably. We have also developed a robust and rotation invariant (in the coronal plane)hand gesture detector using a convolutional neural network. At run time (after having been trained)the detector does not require any pre-processing of the hand images. A complete pipeline for skeleton extraction and hand gesture recognition is developed and employed for real-time physical human-robot interaction, demonstrating the promising capability of the designed framework. This work establishes a firm basis and will be extended for the development of intelligent human intention detection in physical human-robot interaction scenarios, to efficiently recognize a variety of static as well as dynamic gestures.

To make production lines more flexible, dual-arm robots are good candidates to be deployed in autonomous assembly units. In this paper, we propose a sparse kinematic control strategy, that minimizes the number of joints actuated for a coordinated task between two arms. The control strategy is based on a hierarchical sparse QP architecture. We present experimental results that highlight the capability of this architecture to produce sparser motions (for an assembly task) than those obtained with standard controllers.

Deforming a cable to a desired (reachable) shape is a trivial task for a human to do without even knowing the internal dynamics of the cable. This paper proposes a framework for cable shapes manipulation with multiple robot manipulators. The shape is parameterized by a Fourier series. A local deformation model of the cable is estimated on-line with the shape parameters. Using the deformation model, a velocity control law is applied on the robot to deform the cable into the desired shape. Experiments on a dual-arm manipulator are conducted to validate the framework.

This paper presents a tentacle-based obstacle avoidance scheme for omnidirectional mobile robots that must satisfy visibility constraints during navigation. The navigation task consists of driving the robot towards a visual target in the presence of environment (static or moving) obstacles. The target is acquired by an on-board camera, while the obstacles surrounding the robot are sensed by laser range scanners. To perform such task, the robot must avoid the obstacles while maintaining the target in its field of view. The approach is validated in both simulated and real experiments.

In this paper, we present a control strategy that enables intuitive physical human-robot collaboration with mobile manipulators equipped with an omnidirectional base. When interacting with a human operator, intuitiveness of operation is a major concern. To this end, we propose a redundancy solution that allows the mobile base to be fixed when working locally and moves it only when the robot approaches a set of constraints. These constraints include distance to singular poses, minimum of manipulability and distance to objects and angular deviation. Experimental results with a Kuka LWR4 arm mounted on a Neobotix MPO700 mobile base validate the proposed approach.

This paper is about the design of humanoid walking pattern generators to be used for physical collaboration. A particular use case is a humanoid robot helping a human to carry large and/or heavy objects. To do this, we construct a reduced model which takes into account physical interaction. This is used in a model predictive control framework to generate separate behaviors for being a follower or a leader. The approach is then validated both on simulation and on the HRP-4 humanoid robot.

In human-robot interaction, the robot must behave safely, especially when an operator is present in its workspace. Even higher safety levels must be attained when physical contact occurs between the two. To this end, standards such as the ISO10218 define the requirements for a robot to be considered safe for interaction with human operators in an industrial environment. In this paper, we propose an adaptive damping controller that fulfills the ISO10218 requirements by limiting the tool velocity, power and contact force online (and only when needed). The controller is experimentally validated on a hand-arm robotic system, in a mock-up collaborative application. For the hand, safe interaction is enhanced by using tactile sensing, both to regulate grasp forces and to provide an intuitive interface for the operator.

Non-holonomic omnidirectional mobile robots have higher load carrying capacity than their holonomic counterparts. Once the steer joint configuration is initialized, they can perform arbitrarily complex three-dimensional trajectories in the plane of motion and, as such, are more suitable for industrial contexts. However, their kinematic model presents representational and structural singularities, solutions to which must respect actuator performance limits. Recent research efforts have provided either simple restricting of the velocity space (among which few considered hardware limits) or complex non-restricting (no hardware limits considered) solutions. Most of these efforts are providing solutions at the kinematic control level. Instead, here we propose both a representational singularity free kinematic model, and a simple numeric treatment for the kinematic singularity. We further provide a method to tune the latter, to respect the actuator acceleration limits. Thanks to its steer rate damping behavior, the method can be further extended, to respect joint limits. Another benefit is the treatment of the singularity at the level of the kinematic model, which enhances real time capabilities. The developed method has been tested successfully on the Neobotix-MPO700 mobile robot and shown superior results as compared to the embedded controller.

This study presents a voxel-based multiple regression analysis of different magnetic resonance image modalities, including anatomical T1-weighted, T2(*) relaxometry, and diffusion tensor imaging. Quantitative parameters sensitive to complementary brain tissue alterations, including morphometric atrophy, mineralization, microstructural damage, and anisotropy loss, were compared in a linear physiological aging model in 140 healthy subjects (range 20-74 years). The performance of different predictors and the identification of the best biomarker of age-induced structural variation were compared without a priori anatomical knowledge. The best quantitative predictors in several brain regions were iron deposition and microstructural damage, rather than macroscopic tissue atrophy. Age variations were best resolved with a combination of markers, suggesting that multiple predictors better capture age-induced tissue alterations. The results of the linear model were used to predict apparent age in different regions of individual brain. This approach pointed to a number of novel applications that could potentially help highlighting areas particularly vulnerable to disease.

This paper presents a framework to embody a user (e.g. disabled persons) into a humanoid robot controlled by means of brain-computer interfaces (BCI). With our framework, the robot can interact with the environment, or assist its user. The low frequency and accuracy of the BCI commands is compensated by vision tools, such as objects recognition and mapping techniques, as well as shared-control approaches. As a result, the proposed framework offers intuitive, safe, and accurate robot navigation towards an object or a person. The generic aspect of the framework is demonstrated by two complex experiments, where the user controls the robot to serve him a drink, and to raise his own arm.

We propose a framework for combining vision and haptic information in human-robot joint actions. It consists of a hybrid controller that uses both visual servoing and impedance controllers. This can be applied to tasks that cannot be done with vision or haptic information alone. In this framework, the state of the task can be obtained from visual information while haptic information is crucial for safe physical interaction with the human partner. The approach is validated on the task of jointly carrying a flat surface (e.g. a table) and then preventing an object (e.g. a ball) on top from falling off. The results show that this task can be successfully achieved. Furthermore, the framework presented allows for a more collaborative setup, by imparting task knowledge to the robot as opposed to a passive follower.

Moving obstacle avoidance is a fundamental requirement for any robot operating in real environments, where pedestrians, bicycles, and cars are present. In this paper, we propose and validate a framework for avoiding moving obstacles during visual navigation with a wheeled mobile robot. Visual navigation consists of following a path, represented as an ordered set of key images, which have been acquired by an on-board camera in a teaching phase. While following such a path, our robot is able to avoid static and moving obstacles, which were not present during teaching, and which are sensed by an on-board lidar. The proposed approach takes explicitly into account obstacle velocities, estimated using an appropriate Kalman-based observer. The velocities are then used to predict the obstacle positions within a tentacle-based approach. Finally, our approach is validated in a series of real outdoor experiments, showing that when the obstacle velocities are considered, the robot behavior is safer, smoother, and faster than when it is not.

Moving obstacle avoidance is a fundamental requirement for any robot operating in real environments, where pedestrians, bicycles and cars are present. In this work, we design and validate a new approach that takes explicitly into account obstacle velocities, to achieve safe visual navigation in outdoor scenarios. A wheeled vehicle, equipped with an actuated pinhole camera and with a lidar, must follow a path represented by key images, without colliding with the obstacles. To estimate the obstacle velocities, we design a Kalman-based observer. Then, we adapt the tentacles designed in [1], to take into account the predicted obstacle positions. Finally, we validate our approach in a series of simulated and real experiments, showing that when the obstacle velocities are considered, the robot behaviour is safer, smoother, and faster than when it is not.

In this paper, a first step is taken towards using vision in human-humanoid haptic joint actions. Haptic joint actions are characterized by physical interaction throughout the execution of a common goal. Because of this, most of the focus is on the use of force/torque-based control. However, force/torque information is not rich enough for some tasks. Here, a particular case is shown: height stabilization during table carrying. To achieve this, a visual servoing controller is used to generate a reference trajectory for the impedance controller. The control law design is fully described along with important considerations for the vision algorithm and a framework to make pose estimation robust during the table carrying task of the humanoid robot. We then demonstrate all this by an experiment where a human and the HRP-2 humanoid jointly transport a beam using combined force and vision data to adjust the interaction impedance while at the same time keeping the inclination of the beam horizontal.

For intuitive human-robot collaboration, the robot must quickly adapt to the human behavior. To this end, we propose a multimodal sensor-based control framework, enabling a robot to recognize human intention, and consequently adapt its control strategy. Our approach is marker-less, relies on a Kinect and on an on-board camera, and is based on a unified task formalism. Moreover, we validate it in a mock-up industrial scenario, where human and robot must collaborate to insert screws in a flank.

Automation of logistics tasks for small lot sizes and flexible production processes requires intuitive and easy-to-use systems that allow non-expert shop floor workers to naturally instruct transportation systems. To this end, we present a novel laser-based scheme for teach-and-repeat of mobile robot trajectories that relies on scan matching to localize the robot relative to a taught trajectory, which is represented by a sequence of raw odometry and 2D laser data. This approach has two advantages. First, it does not require to build a globally consistent metrical map of the environment, which reduces setup time. Second, the direct use of raw sensor data avoids additional errors that might be introduced by the fact that grid maps only provide an approximation of the environment. Real-world experiments carried out with a holonomic and a differential drive platform demonstrate that our approach repeats trajectories with an accuracy of a few millimeters. A comparison with a standard Monte Carlo localization approach on grid maps furthermore reveals that our method yields lower tracking errors for teach-and-repeat tasks.

In this paper, we design and validate a new tentacle-based approach, for avoiding obstacles during appearance-based navigation with a wheeled mobile robot. In the past, we have developed a framework for safe visual navigation. The robot follows a path represented as a set of key images, and during obstacle circumnavigation, the on-board camera is actuated to maintain scene visibility. In those works, the model used for obstacle avoidance was obtained using a potential vector field. Here, a more sophisticated and efficient method, that exploits the robot kinematic model, and predicts collision at look-ahead distances, is designed and integrated in that framework. Outdoor experiments comparing the two models show that the new approach presents many advantages. Higher speeds and precision can be attained, very cluttered scenarios involving large obstacles can be successfully dealt with, and the control inputs are smoother.

We present and validate a framework for visual navigation with obstacle avoidance. The approach was originally designed in [1], but major improvements and real outdoor experiments are added here. Visual navigation consists of following a path, represented as an ordered set of key images, that have been acquired in a preliminary teaching phase. While following such path, the robot is able to avoid new obstacles which were not present during teaching, and which are sensed by a range scanner. We guarantee that collision avoidance and navigation are achieved simultaneously by actuating the camera pan angle, in the presence of obstacles, to maintain scene visibility as the robot circumnavigates the obstacle. The circumnavigation verse and the collision risk are estimated using a potential vector field derived from an occupancy grid. The framework can also deal with unavoidable obstacles, which make the robot decelerate and eventually stop.

In this paper, we propose a framework for visual navigation with simultaneous obstacle avoidance. The obstacles are modeled by using a vortex potential field, derived from an occupancy grid. Kinematic redundancy guarantees that obstacle avoidance and navigation are achieved concurrently, and the whole scheme is merely sensor-based. The problem is solved both in an obstacle-free and in a dangerous context, and the control law is smoothened in the intermediate situations. In a series of simulations, we show that with our framework, a robot can replay a taught visual path while avoiding collisions, even in the presence of visual occlusions.

In this paper, we present a controller for visual navigation, which utilizes a time-independent varying reference in the feedback law. The navigation framework relies on a monocular camera, and the path is represented as a series of key images. The varying reference is determined using a vector field, derived from the previous and next key images. Results in a simulated environment, as well as on a real robot, show the advantages of the varying reference, with respect to a fixed one, in the image, as well as in the 3D state space.

A catadioptric vision system combines a camera and a mirror to achieve a wide field of view imaging system. This type of vision system has many potential applications in mobile robotics. This paper is concerned with the design of a robust image-based control scheme using a catadioptric vision system mounted on a mobile robot. We exploit the fact that the decoupling property contributes to the robustness of a control method. More precisely, from the image of a point, we propose a minimal and decoupled set of features measurable on any catadioptric vision system. Using the minimal set, a classical control method is proved to be robust in the presence of point range errors. Finally, experimental results with a coarsely calibrated mobile robot validate the robustness of the new decoupled scheme.

We present an image-based visual servoing controller enabling nonholonomic mobile robots with a fixed pinhole camera to reach and follow a continuous path on the ground. The controller utilizes only a small set of features extracted from the image plane, without using the complete geometric representation of the path. A Lyapunov-based stability analysis is carried out. The performance of the controller is validated and compared by simulations and experiments on a car-like robot equipped with a pinhole camera.

We present a visual servoing scheme enabling non-holonomic mobile robots with a fixed pinhole camera to reach and follow a continuous path on the ground. The controller utilizes only a small set of features extracted from the image plane, without using the complete geometric representation of the path. The scheme is position-based, and a Lyapunov-based stability analysis is carried out. The performance of our control design is experimentally validated on a car-like robot equipped with a pinhole camera.

In real-world robotic applications, many factors, both at low-level (e.g., vision and motion control parameters) and at high-level (e.g., the behaviors) determine the quality of the robot performance. Thus, for many tasks, robots require fine tuning of the parameters, in the implementation of behaviors and basic control actions, as well as in strategic decisional processes. In recent years, machine learning techniques have been used to find optimal parameter sets for different behaviors. However, a drawback of learning techniques is time consumption: in practical applications, methods designed for physical robots must be effective with small amounts of data. In this paper, we present a method for concurrent learning of best strategy and optimal parameters, by extending the policy gradient reinforcement learning algorithm. The results of our experimental work in a simulated environment and on a real robot show a very high convergence rate.