Haptic robotic exploration aims to control the movements of a robot with the objective of touching an object and retrieving physical information about it. In this work, we present an innovative exploration strategy to simultaneously detect symmetries in a 3-D object and use this information to enhance shape estimation. This is achieved by leveraging a novel formulation of Gaussian process models that allows the modeling of symmetric surfaces. Our procedure does not assume any prior knowledge about the object, neither about its shape nor about the presence and type of symmetry, necessitating only an approximate estimate of the size and boundaries (bounding box). We report experimental results both in simulation and in the real world, showing that using symmetric models leads to a reduction in shape estimation error, exploration time, and in the number of physical contacts performed by a robot when exploring objects that have symmetries.

This work aims to tackle the intent recognition problem in Human-Robot Collaborative assembly scenarios. Precisely, we consider an interactive assembly of a wooden stool where the robot fetches the pieces in the correct order and the human builds the parts following the instruction manual. The intent recognition is limited to the idle state estimation and it is needed to ensure a better synchronization between the two agents. We carried out a comparison between two distinct solutions involving wearable sensors and eye tracking integrated into the perception pipeline of a flexible planning architecture based on Hierarchical Task Networks. At runtime, the wearable sensing module exploits the raw measurements from four 9-axis Inertial Measurement Units positioned on the wrists and hands of the user as an input for a Long Short-Term Memory Network. On the other hand, the eye tracking relies on a Head Mounted Display and Unreal Engine.

This paper presents a comprehensive framework to enhance Human-Robot Collaboration (HRC) in real-world scenarios. It introduces a formalism to model articulated tasks, requiring cooperation between two agents, through a smaller set of primitives. Our implementation leverages Hierarchical Task Networks (HTN) planning and a modular multisensory perception pipeline, which includes vision, human activity recognition, and tactile sensing. To showcase the system’s scalability, we present an experimental scenario where two humans alternate in collaborating with a Baxter robot to assemble four pieces of furniture with variable components. This integration highlights promising advancements in HRC, suggesting a scalable approach for complex, cooperative tasks across diverse applications.

Dexterous in-hand manipulation is a unique and valuable human skill requiring sophisticated sensorimotor interaction with the environment while respecting stability constraints. Satisfying these constraints with generated motions is essential for a robotic platform to achieve reliable in-hand manipulation skills. Explicitly modelling these constraints can be challenging, but they can be implicitly modelled and learned through experience or human demonstrations. We propose a learning and control approach based on dictionaries of motion primitives generated from human demonstrations. To achieve this, we defined an optimization process that combines motion primitives to generate robot fingertip trajectories for moving an object from an initial to a desired final pose. Based on our experiments, our approach allows a robotic hand to handle objects like humans, adhering to stability constraints without requiring explicit formalization. In other words, the proposed motion primitive dictionaries learn and implicitly embed the constraints crucial to the in-hand manipulation task.

As collaborative robots become more common in manufacturing scenarios and adopted in hybrid human-robot teams, we should develop new interaction and communication strategies to ensure smooth collaboration between agents. In this paper, we propose a novel communicative interface that uses Mixed Reality as a medium to perform Kinesthetic Teaching (KT) on any robotic platform. We evaluate our proposed approach in a user study involving multiple subjects and two different robots, comparing traditional physical KT with holographic-based KT through user experience questionnaires and task-related metrics.

This article investigates mixed reality (MR) to enhance human-robot collaboration (HRC). The proposed solution adopts MR as a communication layer to convey a mobile manipulator’s intentions and upcoming actions to the humans with whom it interacts, thus improving their collaboration. A user study involving 20 participants demonstrated the effectiveness of this MR-focused approach in facilitating collaborative tasks, with a positive effect on overall collaboration performances and human satisfaction.

Nowadays, the most adopted technique to address infertility problems is in vitro fertilisation (IVF). However, its success rate is limited, and the associated procedures, known as assisted reproduction technology (ART), suffer from a lack of objectivity at the laboratory level and in clinical practice. This paper deals with applications of Artificial Intelligence (AI) techniques to IVF procedures. Artificial intelligence is considered a promising tool for ascertaining the quality of embryos, a critical step in IVF. Since the oocyte quality influences the final embryo quality, we present a systematic review of the literature on AI-based techniques used to assess oocyte quality; we analyse its results and discuss several promising research directions. In particular, we highlight how AI-based techniques can support the IVF process and examine their current applications as presented in the literature. Then, we discuss the challenges research must face in fully deploying AI-based solutions in current medical practice. Among them, the availability of high-quality data sets as well as standardised imaging protocols and data formats, the use of physics-informed simulation and machine learning techniques, the study of informative, descriptive yet observable features, and, above all, studies of the quality of oocytes and embryos, specifically about their live birth potential. An improved understanding of determinants for oocyte quality can improve success rates while reducing costs, risks for long-term embryo cultures, and bioethical concerns.

Hands are a fundamental tool humans use to interact with the environment and objects. Through hand motions, we can obtain information about the shape and materials of the surfaces we touch, modify our surroundings by interacting with objects, manipulate objects and tools, or communicate with other people by leveraging the power of gestures. For these reasons, sensorized gloves, which can collect information about hand motions and interactions, have been of interest since the 1980s in various fields, such as human–machine interaction and the analysis and control of human motions. Over the last 40 years, research in this field explored different technological approaches and contributed to the popularity of wearable custom and commercial products targeting hand sensorization. Despite a positive research trend, these instruments are not widespread yet outside research environments and devices aimed at research are often ad hoc solutions with a low chance of being reused. This article aims to provide a systematic literature review for custom gloves to analyze their main characteristics and critical issues, from the type and number of sensors to the limitations due to device encumbrance. The collection of this information lays the foundation for a standardization process necessary for future breakthroughs in this research field.

In this paper, we propose a new vision system for marine rescue drones called Falcon. Our system is designed to track and monitor individuals in need of rescue using a wide-angle field of view and a gimbal mechanism to control the camera direction. Additionally, our system utilizes a high-resolution central field of view to assess the priority level of the person in need of rescue. We conducted two experiments to test our system's performance. The first experiment focused on the correlation between the data compression rate of the system and the precision and recall scores of human detection using YOLO in three different flight altitudes. The second experiment examined the correlation between the distance from the center of the images captured by the system and the confidence score of the detected persons, considering the resolution difference from the central region to the periphery.

Implicit communication plays such a crucial role during social exchanges that it must be considered for a good experience in human-robot interaction. This work addresses implicit communication associated with the detection of physical properties, transport, and manipulation of objects. We propose an ecological approach to infer object characteristics from subtle modulations of the natural kinematics occurring during human object manipulation. Similarly, we take inspiration from human strategies to shape robot movements to be communica-tive of the object properties while pursuing the action goals. In a realistic HRI scenario, participants handed over cups - filled with water or empty - to a robotic manipulator that sorted them. We implemented an online classifier to differentiate careful/not careful human movements, associated with the cups' content. We compared our proposed “expressive” controller, which modulates the movements according to the cup filling, against a neutral motion controller. Results show that human kinematics is adjusted during the task, as a function of the cup content, even in reach-to-grasp motion. Moreover, the carefulness during the handover of full cups can be reliably inferred online, well before action completion. Finally, although questionnaires did not reveal explicit preferences from partici-pants, the expressive robot condition improved task efficiency.

This article presents an open-source architecture for conveying robots’ intentions to human teammates using Mixed Reality and Head-Mounted Displays. The architecture has been developed focusing on its modularity and re-usability aspects. Both binaries and source code are available, enabling researchers and companies to adopt the proposed architecture as a standalone solution or to integrate it in more comprehensive implementations. Due to its scalability, the proposed architecture can be easily employed to develop shared Mixed Reality experiences involving multiple robots and human teammates in complex collaborative scenarios.

The possibility for humans to interact with physical or virtual systems using gestures has been vastly explored by researchers and designers in the last 20 years to provide new and intuitive interaction modalities. Unfortunately, the literature about gestural interaction is not homogeneous, and it is characterized by a lack of shared terminology. This leads to fragmented results and makes it difficult for research activities to build on top of state-of-the-art results and approaches. The analysis in this article aims at creating a common conceptual design framework to enforce development efforts in gesture-based human-machine interaction (HMI). The main contributions of this article can be summarized as follows: 1) we provide a broad definition for the notion of functional gesture in HMI; 2) we design a flexible and expandable gesture taxonomy; and 3) we put forward a detailed problem statement for gesture-based HMI. Finally, to support our main contribution, this article presents and analyzes 83 most pertinent articles classified on the basis of our taxonomy and problem statement.

Humans engaged in collaborative activities are naturally able to convey their intentions to teammates through multi-modal communication, which is made up of explicit and implicit cues. Similarly, a more natural form of human-robot collaboration may be achieved by enabling robots to convey their intentions to human teammates via multiple communication channels. In this paper, we postulate that a better communication may take place should collaborative robots be able to anticipate their movements to human teammates in an intuitive way. In order to support such a claim, we propose a robot system's architecture through which robots can communicate planned motions to human teammates leveraging a Mixed Reality interface powered by modern head-mounted displays. Specifically, the robot's hologram, which is superimposed to the real robot in the human teammate's point of view, shows the robot's future movements, allowing the human to understand them in advance, and possibly react to them in an appropriate way. We conduct a preliminary user study to evaluate the effectiveness of the proposed anticipatory visualization during a complex collaborative task. The experimental results suggest that an improved and more natural collaboration can be achieved by employing this anticipatory communication mode.

Even though cobots have high potential in bringing several benefits in the manufacturing and logistic processes, their rapid (re-)deployment in changing environments is still limited. To enable fast adaptation to new product demands and to boost the fitness of the human workers to the allocated tasks, we propose a novel method that optimizes assembly strategies and distributes the effort among the workers in human-robot cooperative tasks. The cooperation model exploits AND/OR Graphs that we adapted to solve also the role allocation problem. The allocation algorithm considers quantitative measurements that are computed online to describe human operators' ergonomic status and task properties. We conducted preliminary experiments to demonstrate that the proposed approach succeeds in controlling the task allocation process to ensure safe and ergonomic conditions for the human worker.

In this paper, the authors present Sidewinder, a unique stereo vision system for snake-shaped robots operating in search and rescue scenarios. The robot should navigate the environment by finding and passing through narrow spaces (i.e., forward vision task) and search for persons in need of rescue (i.e., panoramic vision task). Therefore, we propose two types of image mapping methods to generate respectively input images for stereo visual SLAM and person detection. This work presents preliminary results using Sidewinder's images as input for the ORS-SLAM2 for localization and mapping, and YOLO-v3, for person detection.

Dexterous in-hand manipulation is a peculiar and useful human skill. This ability requires the coordination of many senses and hand motion to adhere to many constraints. These constraints vary and can be influenced by the object characteristics or the specific application. One of the key elements for a robotic platform to implement reliable in-hand manipulation skills is to be able to integrate those constraints in their motion generations. These constraints can be implicitly modelled, learned through experience or human demonstrations. We propose a method based on motion primitives dictionaries to learn and reproduce in-hand manipulation skills. In particular, we focused on fingertip motions during the manipulation, and we defined an optimization process to combine motion primitives to reach specific fingertip configurations. The results of this work show that the proposed approach can generate manipulation motion coherent with the human one and that manipulation constraints are inherited even without an explicit formalization.

This paper presents a method of wide angle fovea visual odometry (WAF-VO) for Wide Angle Fovea Sensor SLAM (WAF-SLAM), by which a unique locally-high accurate and wide-angle map is generated in addition to camera motion estimation. The WAF sensor is a special-made wide-angle sensor that is inspired from human visual function, i.e., the spatial resolution of the image is not uniform throughout the entire field of view (FOV); it is much higher in the central FOV and decreases rapidly towards the periphery. Our visual odometry method is strongly characterized by a wide-angle FOV and space-variant resolution of the input image from the WAF sensor. A locally-high accurate and wide-angle mapping method is proposed as a major part for WAF-SLAM together with the camera motion estimation. Our proposed method estimates camera motions more stably using very low-spatial-resolution wide-angle images remapped from the input image of the WAF sensor. Using the estimated camera motions, narrow-angle high accurate maps are generated from corresponding feature points in high-spatial resolution central regions of the input image. Wide-angle maps are generated from ones in middle-spatial-resolution wide-angle images remapped from the input image apart from the above very low-spatial-resolution images. When the wide-angle maps are generated, the number of extracted feature points is increased by adjusting contrast threshold values of SIFT feature according to regions of the FOV. A KNN matching method improved using epipolar constraint is proposed and employed for avoidance of mismatching the increased feature points. Thus, the wide-angle maps are generated from more correct corresponding feature points. Finally, the above two types of maps are combined into the unique locally-high accurate and wide-angle map, i.e., a WAF map. Using our proposed method, the WAF map was generated by verification experiments. Furthermore, the paper presents an evaluation of the accuracy and precision of the generated map.

In this work, we present a framework for human-robot collaboration allowing the human operator to alter the robot plan execution online. To achieve this goal, we introduce Branched AND/OR graphs, an extension to AND/OR graphs, to manage flexible and adaptable human-robot collaboration. In our study, the operator can alter the plan execution using two implementations of Branched AND/OR graphs for learning by demonstration, using kinesthetic teaching, and task repetition. Finally, we demonstrated the effectiveness of our framework in a defect spotting scenario where the operator supervises robot operations and modifies online the plan when necessary.

Gestures are a natural communication modality for humans. The ability to interpret gestures is fundamental for robots aiming to naturally interact with humans. Wearable sensors are promising to monitor human activity, in particular the usage of triaxial accelerometers for gesture recognition have been explored. Despite this, the state of the art presents lack of systems for reliable online gesture recognition using accelerometer data. The article proposes SLOTH, an architecture for online gesture recognition, based on a wearable triaxial accelerometer, a Recurrent Neural Network (RNN) probabilistic classifier and a procedure for continuous gesture detection, relying on modelling gesture probabilities, that guarantees (i) good recognition results in terms of precision and recall, (ii) immediate system reactivity.