Effective execution of long-horizon tasks with dexterous robotic hands remains a significant challenge in real-world problems. While learning from human demonstrations has shown encouraging results, they require extensive data collection for training. Hence, decomposing long-horizon tasks into reusable primitive skills is a more efficient approach. To achieve so, we developed DexSkills, a novel supervised learning framework that addresses long-horizon dexterous manipulation tasks using primitive skills. DexSkills is trained to recognize and replicate a select set of skills using human demonstration data, which can then segment a demonstrated long-horizon dexterous manipulation task into a sequence of primitive skills to achieve one-shot execution by the robot directly. Significantly, DexSkills operates solely on proprioceptive and tactile data, i.e., haptic data. Our real-world robotic experiments show that DexSkills can accurately segment skills, thereby enabling autonomous robot execution of a diverse range of tasks.

Distributed sensor arrays capable of detecting multiple spatially distributed stimuli are considered an important element in the realisation of exteroceptive and proprioceptive soft robots. This paper expands upon the previously presented idea of decoupling the measurements of pressure and location of a local indentation from global deformation, using the overall stretch experienced by a soft capacitive e-skin. We employed machine learning methods to decouple and predict these highly coupled deformation stimuli, collecting data from a soft sensor e-skin which was then fed to a machine learning system comprising of linear regressor, gaussian process regressor, SVM and random forest classifier for stretch, force, detection and localisation respectively. We also studied how the localisation and forces are affected when two forces are applied simultaneously. Soft sensor arrays aided by appropriately chosen machine learning techniques can pave the way to e-skins capable of deciphering multi-modal stimuli in soft robots.

Using robot manipulators in contexts where it is undesirable or impractical for humans to physically intervene is crucial for several applications, from manufacturing to extreme environments. However, robots require a high degree of intelligence to operate in those environments, especially if they are not fully structured. Teleoperation compensates for this limitation by connecting the human operator to the robot using human-robot interfaces. The remotely operated sessions can also be used as demonstrations to program more powerful autonomous agents. In this article, we report a thorough user study to characterise the effect of simple vibrotactile feedback on the performance and cognitive load of the human user in performing teleoperated grasping and manipulation tasks. The experiments are performed using a portable and affordable bilateral teleoperation system that we designed, composed of a Leap Motion sensor and a custom-designed vibrotactile haptic glove to operate a 4-fingered robot hand equipped with 3-axis force sensors on the fingertips; the software packages we developed are open-source and publicly available. Our results show that vibrotactile feedback improves teleoperation and reduces cognitive load, especially for complex in-hand manipulation tasks.

The ability to detect that a grasped object is slipping from the robot gripper is a crucial skill for autonomous robotic manipulation. However, current solutions for automatic slip detection do not perform well in real-world unstructured settings, in which a wide variety of gripper-object interactions could occur. Tactile and force sensing are the most suitable sensory modalities to detect such events, and the recent technological advances in the field are generating novel interesting opportunities. In this work, we propose a data-driven method for automatic slip detection that leverages a novel sensor, which combines the advantages of tactile and force sensing, i.e. distributed measurements of normal and shear contact forces. Interestingly, our model is trained (and tested) uniquely with data obtained during routine robot operations (i.e. in the wild) rather than during a controlled data collection procedure. We compare different sets of tactile/force features to highlight the advantages provided by the different sensory modalities, and we report results that show good detection performances on our in-the-wild dataset, which we make publicly available.

We present a system for temporal detection of social interactions. Many of the works until now have succeeded in recognising activities from clipped videos in datasets, but for robotic applications, it is important to be able to move to more realistic data. For this reason, the proposed approach temporally detects intervals where individual or social activity is occurring. Recognition of human activities is a key feature for analysing the human behaviour. In particular, recognition of social activities is useful to trigger human-robot interactions or to detect situations of potential danger. Based on that, this research has three goals: (1) define a new set of descriptors, which are able to characterise human interactions; (2) develop a computational model to segment temporal intervals with social interaction or individual behaviour; (3) provide a public dataset with RGB-D data with continuous stream of individual activities and social interactions. Results show that the proposed approach attained relevant performance with temporal segmentation of social activities.

We investigate how incremental learning of long-term human activity patterns improves the accuracy of activity classification over time. Rather than trying to improve the classification methods themselves, we assume that they can take into account prior probabilities of activities occurring at a particular time. We use the classification results to build temporal models that can provide these priors to the classifiers. As our system gradually learns about typical patterns of human activities, the accuracy of activity classification improves, which results in even more accurate priors. Two datasets collected over several months containing hand-annotated activity in residential and office environments were chosen to evaluate the approach. Several types of temporal models were evaluated for each of these datasets. The results indicate that incremental learning of daily routines leads to a significant improvement in activity classification.

Social activity based on body motion is a key feature for non-verbal and physical behavior defined as function for communicative signal and social interaction between individuals. Social activity recognition is important to study human-human communication and also human-robot interaction. Based on that, this research has threefold goals: (1) recognition of social behavior (e.g. human-human interaction) using a probabilistic approach that merges spatio-temporal features from individual bodies and social features from the relationship between two individuals; (2) learn priors based on physical proximity between individuals during an interaction using proxemics theory to feed a probabilistic ensemble of activity classifiers; and (3) provide a public dataset with RGB-D data of social daily activities including risk situations useful to test approaches for assisted living, since this type of dataset is still missing. Results show that using the proposed approach designed to merge features with different semantics and proximity priors improves the classification performance in terms of precision, recall and accuracy when compared with other approaches that employ alternative strategies.