Haptic robotic exploration aims to control the movements of a robot with the objective of touching an object and retrieving physical information about it. In this work, we present an innovative exploration strategy to simultaneously detect symmetries in a 3-D object and use this information to enhance shape estimation. This is achieved by leveraging a novel formulation of Gaussian process models that allows the modeling of symmetric surfaces. Our procedure does not assume any prior knowledge about the object, neither about its shape nor about the presence and type of symmetry, necessitating only an approximate estimate of the size and boundaries (bounding box). We report experimental results both in simulation and in the real world, showing that using symmetric models leads to a reduction in shape estimation error, exploration time, and in the number of physical contacts performed by a robot when exploring objects that have symmetries.

While visual sensing is often the predominant modality for a robot to localize objects in the environment, tactile and force sensing become crucial when objects are occluded, poorly visible, or buried. However, existing works on locating buried objects rely solely on force measurements at a single contact point on the robot end-effector, making 3D localization very challenging. This paper presents an alternative approach using a tactile sensor that measures both normal and shear forces (i.e. 3-axis) on distributed points; three Long Short-Term Memory (LSTM) models are trained with real-world data to perform real-time 3D localization (i.e. distance, direction and depth) of an object buried within a granular material. Our experimental results suggest that measuring both normal and shear forces (instead of just normal) on distributed contact points (instead of only one point) is essential for the accurate 3D localization of buried objects.

Effective execution of long-horizon tasks with dexterous robotic hands remains a significant challenge in real-world problems. While learning from human demonstrations has shown encouraging results, they require extensive data collection for training. Hence, decomposing long-horizon tasks into reusable primitive skills is a more efficient approach. To achieve so, we developed DexSkills, a novel supervised learning framework that addresses long-horizon dexterous manipulation tasks using primitive skills. DexSkills is trained to recognize and replicate a select set of skills using human demonstration data, which can then segment a demonstrated long-horizon dexterous manipulation task into a sequence of primitive skills to achieve one-shot execution by the robot directly. Significantly, DexSkills operates solely on proprioceptive and tactile data, i.e., haptic data. Our real-world robotic experiments show that DexSkills can accurately segment skills, thereby enabling autonomous robot execution of a diverse range of tasks.

Multifingered robot hands can be extremely effective in physically exploring and recognizing objects, especially if they are extensively covered with distributed tactile sensors. Convolutional neural networks (CNNs) have been proven successful in processing high dimensional data, such as camera images, and are, therefore, very well suited to analyze distributed tactile information as well. However, a major challenge is to organize tactile inputs coming from different locations on the hand in a coherent structure that could leverage the computational properties of the CNN. Therefore, we introduce a morphology-specific CNN (MS-CNN), in which hierarchical convolutional layers are formed following the physical configuration of the tactile sensors on the robot. We equipped a four-fingered Allegro robot hand with several uSkin tactile sensors; overall, the hand is covered with 240 sensitive elements, each one measuring three-axis contact force. The MS-CNN layers process the tactile data hierarchically: at the level of small local clusters first, then each finger, and then the entire hand. We show experimentally that, after training, the robot hand can successfully recognize objects by a single touch, with a recognition rate of over 95%. Interestingly, the learned MS-CNN representation transfers well to novel tasks: by adding a limited amount of data about new objects, the network can recognize nine types of physical properties.

Robotic grasping is fundamental to many real-world applications, and new approaches must be systematically evaluated. However, in most cases, the performance of a specific approach is assessed by simply counting the number of successful attempts in a given task, and this success rate is then compared to those of other solutions, without taking into account the random variability across different experiments (e.g. due to sensor noise or variations in object placement). In order to address this issue, we classify the observed performance into qualitatively ordered outcomes, thereby stratifying the results. We then show how to analyze these results in a statistical framework, which accounts for the variability between experiments. The advantages of our approach are demonstrated in the practical comparison of four grasp planning algorithms. In particular, we show that the proposed approach allows us to carry out several distinct evaluations from a single set of experiments, without having to repeat the data collection process. We demonstrate that differences between the algorithms, which would not be apparent from overall success rates, can be identified and evaluated.

The shapes of most real-world objects are symmetric with respect to at least one plane of symmetry. This information is unconsciously used by humans when they attempt to estimate the shape of an object in presence of uncertainty or missing evidence, for example if the object is partially occluded or if they are exploring the object by touch (i.e. haptic exploration). In robotics, this concept has been used for the visual estimation of object shape. However, no attempt has been made so far to incorporate this idea into haptic-based estimation. This work presents a method for the haptic exploration of object shape that includes the assumption that a symmetry could exist. The approach combines a tailored version of Gaussian Processes and a novel exploratory procedure that is able to detect the position and orientation of any plane of symmetry. Our results show that, if one or more symmetries exist, the object shape can be estimated faster and more accurately. Interestingly, in the case that no symmetry is present, the exploration process is only slightly slower, and the final accuracy of the shape estimation is not compromised.

Using robot manipulators in contexts where it is undesirable or impractical for humans to physically intervene is crucial for several applications, from manufacturing to extreme environments. However, robots require a high degree of intelligence to operate in those environments, especially if they are not fully structured. Teleoperation compensates for this limitation by connecting the human operator to the robot using human-robot interfaces. The remotely operated sessions can also be used as demonstrations to program more powerful autonomous agents. In this article, we report a thorough user study to characterise the effect of simple vibrotactile feedback on the performance and cognitive load of the human user in performing teleoperated grasping and manipulation tasks. The experiments are performed using a portable and affordable bilateral teleoperation system that we designed, composed of a Leap Motion sensor and a custom-designed vibrotactile haptic glove to operate a 4-fingered robot hand equipped with 3-axis force sensors on the fingertips; the software packages we developed are open-source and publicly available. Our results show that vibrotactile feedback improves teleoperation and reduces cognitive load, especially for complex in-hand manipulation tasks.

Long-term and culture-specific experience of music shapes rhythm perception, leading to enculturated expectations that make certain rhythms easier to track and more conducive to synchronized movement. However, the influence of enculturated bias on the moment-to-moment dynamics of rhythm tracking is not well understood. Recent modeling work has formulated entrainment to rhythms as a formal inference problem, where phase is continuously estimated based on precise event times and their correspondence to timing expectations: PIPPET (Phase Inference from Point Process Event Timing). Here we propose that the problem of optimally tracking a rhythm also requires an ongoing process of inferring which pattern of event timing expectations is most suitable to predict a stimulus rhythm. We formalize this insight as an extension of PIPPET called pPIPPET (PIPPET with pattern inference). The variational solution to this problem introduces terms representing the likelihood that a stimulus is based on a particular member of a set of event timing patterns, which we initialize according to culturally-learned prior expectations of a listener. We evaluate pPIPPET in three experiments. First, we demonstrate that pPIPPET can qualitatively reproduce enculturated bias observed in human tapping data for simple two-interval rhythms. Second, we simulate categorization of a continuous three-interval rhythm space by Western-trained musicians through derivation of a comprehensive set of priors for pPIPPET from metrical patterns in a sample of Western rhythms. Third, we simulate iterated reproduction of three-interval rhythms, and show that models configured with notated rhythms from different cultures exhibit both universal and enculturated biases as observed experimentally in listeners from those cultures. These results suggest the influence of enculturated timing expectations on human perceptual and motor entrainment can be understood as approximating optimal inference about the rhythmic stimulus, with respect to prototypical patterns in an empirical sample of rhythms that represent the music-cultural environment of the listener.

The ability to detect that a grasped object is slipping from the robot gripper is a crucial skill for autonomous robotic manipulation. However, current solutions for automatic slip detection do not perform well in real-world unstructured settings, in which a wide variety of gripper-object interactions could occur. Tactile and force sensing are the most suitable sensory modalities to detect such events, and the recent technological advances in the field are generating novel interesting opportunities. In this work, we propose a data-driven method for automatic slip detection that leverages a novel sensor, which combines the advantages of tactile and force sensing, i.e. distributed measurements of normal and shear contact forces. Interestingly, our model is trained (and tested) uniquely with data obtained during routine robot operations (i.e. in the wild) rather than during a controlled data collection procedure. We compare different sets of tactile/force features to highlight the advantages provided by the different sensory modalities, and we report results that show good detection performances on our in-the-wild dataset, which we make publicly available.

Texture sensing is one of the types of information sensed by humans through touch, and is thus of interest to robotics that this type of information can be acquired and processed. In this work we present a texture topography sensor based on a ciliary structure, a biological structure found in many organisms. The device consists of up to 9 elastic cilia with permanent magnetization assembled on top of a highly sensitive tunneling magnetoresistance (TMR) sensor, within a compact footprint of 6×6 mm2. When these cilia brush against some textured surface, their movement and vibrations give rise to a signal that can be correlated to the characteristics of the texture being measured. We also present an electronic signal acquisition board, used in this work. Various configurations of cilia sizes are tested, with the most precise being capable of differentiating different types of sandpaper from 9.2 μm to 213 μm average surface roughness with a 7 μm resolution. As a topography scanner the sensor was able to scan a 20 μm high step in a flat surface.

The growing consumer demand for large volumes of high quality fruit has generated an increasing need for auto-mated fruit quality control during production. Optical methods have been proved successful in a few cases, but with limitations related to the variability of fruit colors and lighting conditions during tests. Tactile sensing provides a valuable alternative, although it comes with the need of a physical interaction that could damage the fruit. To overcome these limitations, we propose the usage of a recently developed soft tactile sensor for non-invasive fruit quality control. The ability of the sensor to detect very small forces and to finely analyze surfaces allows the collection of relevant information about the fruit by performing a very delicate physical interaction, that does not cause any damage. We report experiments in which such information is used to determine whether apples and strawberries are ripe or senescent. We test different configurations of the sensor and different classification algorithms, achieving very good accuracy for both apples (96%) and strawberries (83%).

This work describes a virtual reality (VR) based robot teleoperation framework which relies on scene visualization from depth cameras and implements human-robot and human-scene interaction gestures. We suggest that mounting a camera on a slave robot's end-effector (an in-hand camera) allows the operator to achieve better visualization of the remote scene and improve task performance. We compared experimentally the operator's ability to understand the remote environment in different visualization modes: single external static camera, in-hand camera, in-hand and external static camera, in-hand camera with OctoMap occupancy mapping. The latter option provided the operator with a better understanding of the remote environment whilst requiring relatively small communication bandwidth. Consequently, we propose suitable grasping methods compatible with the VR based teleoperation with the in-hand camera. Video demonstration: https://youtu.be/3vZaEykMS_E.

Dexterous robotic manipulation of unknown objects can open the way to novel tasks and applications of robots in semi-structured and unstructured settings, from advanced industrial manufacturing to exploration of harsh environments. However, it is challenging for at least three reasons: the desired motion of the object might be too complex to be described analytically, precise models of the manipulated objects are not available, the controller should simultaneously ensure both a robust grasp and an effective in-hand motion. To solve these issues we propose to learn in-hand robotic manipulation tasks from human demonstrations, using Dynamical Movement Primitives (DMPs), and to reproduce them with a robust compliant controller based on the Virtual Springs Framework (VSF), that employs real-time feedback of the contact forces measured on the robot fingertips. With this solution, the generalization capabilities of DMPs can be transferred successfully to the dexterous in-hand manipulation problem: we demonstrate this by presenting real-world experiments of in-hand translation and rotation of unknown objects.

Close human-robot cooperation is a key enabler for new developments in advanced manufacturing and assistive applications. Close cooperation require robots that can predict human actions and intent, understanding human non-verbal cues. Recent approaches based on neural networks have led to encouraging results in the human action prediction problem both in continuous and discrete spaces. Our approach extends the research in this direction. Our contributions are three-fold. First, we validate the use of gaze and body pose cues as a means of predicting human action through a feature selection method. Next, we address two shortcomings of existing literature: predicting multiple and variable-length action sequences. This is achieved by applying an encoder-decoder recurrent neural network topology in the discrete action prediction problem. In addition, we theoretically demonstrate the importance of predicting multiple action sequences as a means of estimating the stochastic reward in a human robot cooperation scenario. Finally, we show the ability to effectively train the prediction model on an action prediction dataset, involving human motion data, and explore the influence of the model's parameters on its performance.

Robust grasping is a major, and still unsolved, problem in robotics. Information about the 3D shape of an object can be obtained either from prior knowledge (e.g., accurate models of known objects or approximate models of familiar objects) or real-time sensing (e.g., partial point clouds of unknown objects) and can be used to identify good potential grasps. However, due to modeling and sensing inaccuracies, local exploration is often needed to refine such grasps and successfully apply them in the real world. The recently proposed unscented Bayesian optimization technique can make such exploration safer by selecting grasps that are robust to uncertainty in the input space (e.g., inaccuracies in the grasp execution). Extending our previous work on 2D optimization, in this paper we propose a 3D haptic exploration strategy that combines unscented Bayesian optimization with a novel collision penalty heuristic to find safe grasps in a very efficient way: while by augmenting the search-space to 3D we are able to find better grasps, the collision penalty heuristic allows us to do so without increasing the number of exploration steps.

Typically, the range and sensitivity of force sensors are determined during production. However, to be able to do both delicate and high-force demanding work, adjustable force sensitivity would be beneficial. The current paper proposes such a sensor by implementing a planar electromagnet above a 3-axis magnetic sensor, separated by soft foam. Furthermore, the sensor has digital output with an integrated microcontroller. The magnetic field strength with varying currents is examined in simulation, and the field changes according to displacements are investigated both in simulation and with the actual sensor. A prototype 3-axis force sensor is implemented and the relationship between the magnetic field change and the corresponding applied force is also investigated. It could be shown that the sensitivity of the sensor to displacements, as well as force, can indeed be adjusted.

In this paper we present a low-cost and easy to fabricate 3-axis tactile sensor based on magnetic technology. The sensor consists in a small magnet immersed in a silicone body with an Hall-effect sensor placed below to detect changes in the magnetic field caused by displacements of the magnet, generated by an external force applied to the silicone body. The use of a 3-axis Hall-effect sensor allows to detect the three components of the force vector, and the proposed design assures high sensitivity, low hysteresis and good repeatability of the measurement: notably, the minimum sensed force is about 0.007N. All components are cheap and easy to retrieve and to assemble; the fabrication process is described in detail and it can be easily replicated by other researchers. Sensors with different geometries have been fabricated, calibrated and successfully integrated in the hand of the human-friendly robot Vizzy. In addition to the sensor characterization and validation, real world experiments of object manipulation are reported, showing proper detection of both normal and shear forces.

Vision-based grasping for humanoid robots is a challenging problem due to a multitude of factors. First, humanoid robots use an “eye-to-hand” kinematics configuration that, on the contrary to the more common “eye-in-hand” configuration, demands a precise estimate of the position of the robot's hand. Second, humanoid robots have a long kinematic chain from the eyes to the hands, prone to accumulate the calibration errors of the kinematics model, which offsets the measured hand-to-object relative pose from the real one. In this paper, we propose a method able to solve these two issues jointly. A robust pose estimation of the robot's hand is achieved via a 3D model-based stereo-vision algorithm, using an edge-based distance transform metric and synthetically generated images of a robot's arm-hand internal computer-graphics model (kinematics and appearance). Then, a particle-based optimization method adapts on-line the robot's internal model to match the real and the synthetically generated images, effectively compensating the kinematics calibration errors. We evaluate the proposed approach using a position-based visual-servoing method on the iCub robot, showing the importance of the continuous visual feedback in humanoid grasping tasks.

This paper addresses the problem of having a robot executing motor tasks requested by a human through spoken language. Verbal instructions do not typically have a one-to-one mapping to robot actions, due to various reasons: economy of spoken language, e.g., one short instruction might indeed correspond to a complex sequence of robot actions, and details about action execution might be omitted; grounding, e.g., some actions might need to be added or adapted due to environmental contingencies; embodiment, e.g., a robot might have different means than the human ones to obtain the goals that the instruction refers to. We propose a general cognitive architecture to deal with these issues, based on three steps: i) language-based semantic reasoning on the instruction (high-level), ii) formulation of goals in robot symbols and probabilistic planning to achieve them (mid-level), iii) action execution (low-level). The description of the mid-level is the main focus of this paper. The robot plans are adapted to the current scenario, perceived in real-time and continuously updated, taking in consideration the robot capabilities, modeled through the concept of affordances: this allows for flexibility and creativity in the task execution. We showcase the performance of the proposed architecture with real world experiments using the iCub humanoid robot, also in the presence of unexpected events and action failures.

The concept of affordances facilitates the encoding of relations between actions and effects in an environment centered around the agent. Such an interpretation has important impacts on several cognitive capabilities and manifestations of intelligence, such as prediction and planning. In this paper, a new framework based on denoising Auto-encoders (dA) is proposed which allows an agent to explore its environment and actively learn the affordances of objects and tools by observing the consequences of acting on them. The dA serves as a unified framework to fuse multi-modal data and retrieve an entire missing modality or a feature within a modality given information about other modalities. This work has two major contributions. First, since training the dA is done in continuous space, there will be no need to discretize the dataset and higher accuracies in inference can be achieved with respect to approaches in which data discretization is required (e.g. Bayesian networks). Second, by fixing the structure of the dA, knowledge can be added incrementally making the architecture particularly useful in online learning scenarios. Evaluation scores of real and simulated robotic experiments show improvements over previous approaches while the new model can be applied in a wider range of domains.

Safe and robust grasping of unknown objects is a major challenge in robotics, which has no general solution yet. A promising approach relies on haptic exploration, where active optimization strategies can be employed to reduce the number of exploration trials. One critical problem is that certain optimal grasps discoverd by the optimization procedure may be very sensitive to small deviations of the parameters from their nominal values: we call these unsafe grasps because small errors during motor execution may turn optimal grasps into bad grasps. To reduce the risk of grasp failure, safe grasps should be favoured. Therefore, we propose a new algorithm, unscented Bayesian optimization, that performs efficient optimization while considering uncertainty in the input space, leading to the discovery of safe optima. The results highlight how our method outperforms the classical Bayesian optimization both in synthetic problems and in realistic robot grasp simulations, finding robust and safe grasps after a few exploration trials.

This paper proposes a novel approach to generate trajectories that generalize given demonstrations according to optimality criteria. By formulating the problem as a quadratic program we can efficiently incorporate constraints to adapt to new desired motion requirements while achieving the main goal of matching the acceleration profile of the demonstration. This makes our method particularly suited for the imitation and generalization of trajectories such as hitting movements, where it is crucial to maintain the dynamic traits of the demonstration while respecting strict requirements for the goals position, velocity and time. Our method draws inspiration from the Dynamical Movement Primitives (DMPs) framework, preserving its desirable properties of flexibility and rejection of disturbances during execution. Moreover, it offers an higher degree of control on the generated solution, allowing for example i) to limit the instantaneous positions, velocities and accelerations during the whole trajectory, and ii) to add intermediate way points that were not present in the demonstration. With current state-of-the-art solvers of quadratic programs, a problem with hundreds of parameters can be solved in tens of milliseconds in a standard computer, allowing practical applications. Our methodology results in trajectories with a very good approximation of the shape traits of the demonstration, with additional flexibility in specifying constraints of the generated trajectory.

In this paper a novel approach to kinematics learning and task space control, under switching contexts, is presented. Such non-stationary contexts may appear in many robotic tasks: in particular, the changing of the context due to the use of tools with different lengths and shapes is herein studied. We model the robot forward kinematics as a multi-valued function, in which different outputs for the same input query are related to actual different hidden contexts. To do that, we employ IMLE, a recent online learning algorithm that fits an infinite mixture of linear experts to the online stream of training data. This algorithm can directly provide multi-valued regression in a online fashion, while having, for classic single-valued regression, a performance comparable to state-of-the-art online learning algorithms. The context varying forward kinematics is learned online through exploration, not relying on any kind of prior knowledge. Using the proposed approach, the robot can dynamically learn how to use different tools, without forgetting the kinematic mappings concerning previously manipulated tools. No information is given about such tool changes to the learning algorithm, nor any assumption is made about the tool kinematics. To our knowledge this is the most general and efficient approach to learning and control under discrete varying contexts. Some experimental results obtained on a high-dimensional simulated humanoid robot provide a strong support to our approach.

This paper describes the implementation in a walking humanoid robot of a mental model, allowing the dynamical change of the emotional state of the robot based on external stimuli; the emotional state affects the robot decisions and behavior, and it is expressed with both facial and whole-body patterns. The mental model is applied to KOBIAN-R, a 65-DoFs whole body humanoid robot designed for human-robot interaction and emotion expression. To evaluate the importance of the proposed system in the framework of human-robot interaction and communication, we conducted a survey by showing videos of the robot behaviors to a group of 30 subjects. The results show that the integration of dynamical emotion expression and locomotion makes the humanoid robot more appealing to humans, as it is perceived as more “favorable” and “useful”, and less “robot-like".

This paper describes the development of a new shank mechanism and mimicking the human-like walking in the horizontal and frontal plane. One of human walking characteristics is that the COM (Center Of Mass) motion in the lateral direction is as small as 30 mm. We assume that it is thanks to the human walking characteristics in the horizontal plane that the step width is as narrow as 90 mm and the foot rotation angle is 12 deg. To mimic these characteristics, we developed a new shank and implemented it in a humanoid robot WABIAN-2RIII. It has a parallel mechanism which mimics the shank's size of human. Thanks to its size almost the same as human's the robot is capable of realizing gait with the narrow step width of 90 mm and the foot rotation angle of 12 deg. We evaluated the performance of the shank using WABIAN-2RIII. The robot could realize stepping in place with lateral displacement of CoM within 34 mm, which is almost as small as that of human.

This paper presents a comparison of open-loop and closed-loop control strategies for tracking a task space trajectory, using redundant robots. We do not assume any knowledge of the analytical forward and inverse kinematics, relying instead on learning these models online, while executing a desired task. Specifically, we employ a recent learning algorithm that allows to learn a probabilistic model from which both the forward and inverse solutions can be obtained, as well as the Jacobian of the kinematics map. Such learned model can then be used to implement both types of control. Moreover, the multi-valued solutions provided by the learned model can be applied to redundant systems in which an infinite number of inverse solutions may exist. We present experiments with a simulated version of the iCub, a highly redundant humanoid robot, in which this learned model is employed to execute both open-loop and closed-loop trajectory control. We show the advantages and drawbacks of both control strategies, and we propose a way to combine them to deal with sensor noise and failures, showing the benefits of using a learning algorithm that can simultaneously provide forward and inverse predictions.

A complex relationship exists between national cultural background and interaction with robots, and many earlier studies have investigated how people from different cultures perceive the inclusion of robots into society. Conversely, very few studies have investigated how robots, speaking and using gestures that belong to a certain national culture, are perceived by humans of different cultural background. The purpose of this work is to prove that humans may better accept a robot that can adapt to their specific national culture. This experiment of Human-Robot Interaction was performed in Egypt. Participants (native Egyptians versus Japanese living in Egypt) were shown two robots greeting them and speaking respectively in Arabic and Japanese, through a simulated video conference. Spontaneous reactions of the human subjects were measured in different ways, and participants completed a questionnaire assessing their preferences and their emotional state. Results suggested that Egyptians prefer the Arabic version of the robot, while they report discomfort when interacting with the Japanese version. These findings confirm the importance of a culture-specific customisation of robots in the context of Human-Robot Interaction.

We describe an interactive learning strategy that enables a humanoid robot to build a representation of its workspace: we call it a Reachable Space Map. The robot learns this map autonomously and online during the execution of goal-directed reaching movements; reaching control is based on kinematic models that are learned online as well. The map can be used to estimate the reachability of a fixated object and to plan preparatory movements (e.g. bending or rotating the waist) that improve the effectiveness of the subsequent reaching action. Three main concepts make our solution innovative with respect to previous works: the use of a gaze-centered motor representation to describe the robot workspace, the primary role of action in building and representing knowledge (i.e. interactive learning), the realization of autonomous online learning. We evaluate our strategy by learning the workspace of a simulated humanoid robot and we show how this knowledge can be exploited to plan and execute complex actions, like whole-body bimanual reaching.

This paper describes the control of a human-like robotic neck actuated with tendons. The controller regulates the length of the tendons to achieve a desired orientation of the neck and at the same time it maintains the tension of the tendons within certain limits. The solution we propose does not use any model of the system, but it relies on online learning of the different Jacobian mappings required by the controller. Learning, data acquisition and control are simultaneous; thus learning is completely autonomous, and purely online. We show that after enough iterations the controller produces straight trajectories in the task space and is able to maintain the tension of the tendons within safe limits.