Employing skin-like tactile sensors on robots enhances both the safety and usability of collaborative robots by adding the capability to detect human contact. Unfortunately, simple binary tactile sensors alone cannot determine the context of the human contact—whether it is a deliberate interaction or an unintended collision that requires safety manoeuvres. Many published methods classify discrete interactions using more advanced tactile sensors or by analysing joint torques. Instead, we propose to augment the intention recognition capabilities of simple binary tactile sensors by adding a robot-mounted camera for human posture analysis. Different interaction characteristics, including touch location, human pose, and gaze direction, are used to train a supervised machine learning algorithm to classify whether a touch is intentional or not with an F1-score of 86%. We demonstrate that multimodal intention recognition is significantly more accurate than monomodal analyses with the collaborative robot Baxter. Furthermore, our method can also continuously monitor interactions that fluidly change between intentional or unintentional by gauging the user’s attention through gaze. If a user stops paying attention mid-task, the proposed intention and attention recognition algorithm can activate safety features to prevent unsafe interactions. We also employ a feature reduction technique that reduces the number of inputs to five to achieve a more generalized low-dimensional classifier. This simplification both reduces the amount of training data required and improves real-world classification accuracy. It also renders the method potentially agnostic to the robot and touch sensor architectures while achieving a high degree of task adaptability.Note to Practitioners—Whenever a user interacts physically with a robot, such as in collaborative manufacturing, the robot may respond to unintended touch inputs from the user. This may be through body collisions or that the user is suddenly distracted and is no longer paying attention to what they are doing. We propose an easy-to-implement method to augment safety of physical human-robot collaboration by determining whether the touch from a user is intentional or not through the use of robot-mounted basic touch sensors and computer vision. The algorithm examines the location of the user’s hands relative to the touched sensors in addition to observing where the user is looking. Machine learning is then used to classify in real-time, with an F1-score of 86%, whether a touch is intentional or not such that the robot can react accordingly. The method is particularly applicable in collaborative manufacturing contexts, but can also be applied anywhere where a user physically interacts with a robot. We demonstrate the utility of the method in enhancing safety during human-robot collaboration through a simulated collaborative manufacturing scenario with the robot Baxter, but the method can easily be adapted to other systems.

We propose a novel performance metric for articulated robots with distributed directional sensors called thesensor observability analysis(SOA). These robot-mounted distributed directional sensors (e.g., joint torque sensors) change their individual sensing directions as the joints move. SOA transforms individual sensor axes in joint space to provide the cumulative sensing quality of these sensors to observe each task-space axis, akin to forward kinematics for sensors. For example, certain joint configurations may align joint torque sensors in such a way that they are unable to observe interaction forces in one or more task-space (e.g., Cartesian) axes. The resultant sensor observability performance metrics can then be used in optimization and in null-space control to avoid sensor observability in singular configurations or to maximize sensor observability in particular directions. We use the specific case of force sensing in serial robot manipulators to showcase the analysis. Parallels are drawn between sensor observability and traditional kinematic manipulability; SOA is shown to be more generalizable in terms of analyzing non-joint-mounted sensors and can potentially be applied to sensor types other than for force sensing. Simulations and experiments using a custom 3-degree of freedom robot and the Baxter robot demonstrate the utility and importance of sensor observability in physical interactions.

In order to properly integrate humanoid robots in real-life situations, they must be able to collaborate with humans in completing tasks. One of these tasks is the cooperative transportation of a heavy object, which has been widely studied in the humanoids literature. However, the proposed methods rely heavily on six-axis force/torque (F/T) sensors at the wrists, which medium-sized or even some full-sized humanoid robots do not have. This paper proposes an observer to overcome the lack of F/T sensors. The observer is then coupled with a simplified dynamic model of the transportation task allowing the humanoid robot to carry out the task in a stable way. The method is tested in simulation using a humanoid robot that does not have F/T sensors, a NAO robot, to demonstrate its performance. These tests pointed out that the proposed method successfully estimated the interaction forces while generating stable walking patterns.

In this paper, we propose a preliminary definition and analysis of the novel concept of sensor observability index. The goal is to analyse and evaluate the performance of distributed directional or axial-based sensors to observe specific axes in task space as a function of joint configuration in serial robot manipulators. For example, joint torque sensors are often used in serial robot manipulators and assumed to be perfectly capable of estimating end effector forces, but certain joint configurations may cause one or more task-space axes to be unobservable as a result of how the joint torque sensors are aligned. The proposed sensor observability provides a method to analyse the quality of the current robot configuration to observe the task space. Parallels are drawn between sensor observability and the traditional kinematic Jacobian for the particular case of joint torque sensors in serial robot manipulators. Although similar information can be retrieved from kinematic analysis of the Jacobian transpose in serial manipulators, sensor observability is shown to be more generalizable in terms of analysing non-joint-mounted sensors and other sensor types. In addition, null-space analysis of the Jacobian transpose is susceptible to false observability singularities. Simulations and experiments using the robot Baxter demonstrate the importance of maintaining proper sensor observability in physical interactions.

Rather than systematically programming joint or task trajectories, having a human physically manipulate the robot for direct adjustments is more intuitive, saves time, and increases usability, especially for nonexperts. Interactive motion generation or repositioning of humanoid robots through direct human-touch manipulation is not an easy task, especially for high-level multijoint maneuvers. We propose a set of design rules for generating intuitive touch semantics called the “two-touch kinematic chain paradigm.” Our method interprets user touch intentions to allow motions ranging from low-level single joint control to high-level whole-body task control with posture generation, stepping, and walking. The goal is to provide the user with an intuitive protocol for physical humanoid manipulation that can serve the purpose of any application. The generated set of touch semantics is embodied in a finite state machine-based framework using a task-space quadratic programming controller to interpret human touch using capacitive sensors embedded in the humanoid shell, and force-torque sensors located at the ankles and wrists. A position-controlled humanoid robot is used to assess the utility and function of our proposed touch semantics for physical manipulation. Furthermore, a user study with nonexperts examines how our approach is perceived in practice.

Micromanipulation of individual biological cells, such as embryos during preimplantation genetic diagnosis (PGD), is a delicate and time-consuming task. Two major procedures in PGD include the rotation of the embryo to gain a more favorable position for zona breaching, and the extraction of a blastomere after an opening has been made in the zona pellucida. Rotation tracking of cleavage-stage embryos has not been reported given their lack of distinctive features. In this manuscript, a geometric model for partially determining the three dimensional (3-D) angular position of 2-cell embryos using two dimensional (2-D) microscopic brightfield images was derived and verified using a computer generated model. This model was then applied using computer vision algorithms on a rotating cleavage-stage mouse embryo, demonstrating partial 3-D rotation tracking. Furthermore, embryo micromanipulation tasks are typically performed manually using micropipettes. Technological advances have made automation of these tasks possible. This manuscript also presents computer vision algorithms for the segmentation and calibration of micropipettes. The calibration procedure allowed automated position control of the micropipettes without the need for real-time vision feedback using micropipette recognition algorithms, and effective position control was verified using semi-automated blastomere extraction experiments. This manuscript presents preliminary work towards the automation of cell manipulation procedures.