The intuitive collaboration of humans and intelligent robots (embodied AI) in the real-world is an essential objective for many desirable applications of robotics. Whilst there is much research regarding explicit communication, we focus on how humans and robots interact implicitly, on motor adaptation level. We present a real-world setup of a human-robot collaborative maze game, designed to be non-trivial and only solvable through collaboration, by limiting the actions to rotations of two orthogonal axes, and assigning each axes to one player. This results in neither the human nor the agent being able to solve the game on their own. We use deep reinforcement learning for the control of the robotic agent, and achieve results within 30 minutes of real-world play, without any type of pre-training. We then use this setup to perform systematic experiments on human/agent behaviour and adaptation when co-learning a policy for the collaborative game. We present results on how co-policy learning occurs over time between the human and the robotic agent resulting in each participant's agent serving as a representation of how they would play the game. This allows us to relate a person's success when playing with different agents than their own, by comparing the policy of the agent with that of their own agent.

This paper describes a novel approach in human-robot interaction driven by ergonomics. With a clear focus on optimising ergonomics, the approach proposed here continuously observes a human user’s posture and by invoking appropriate cooperative robot movements, the user’s posture is, whenever required, brought back to an ergonomic optimum. Effectively, the new protocol optimises the human-robot relative position and orientation as a function of human ergonomics. An RGB-D camera is used to calculate and monitor human joint angles in real-time and to determine the current ergonomics state. A total of 6 main causes of low ergonomic states are identified, leading to 6 universal robot responses to allow the human to return to an optimal ergonomics state. The algorithmic framework identifies these 6 causes and controls the cooperating robot to always adapt the environment (e.g. change the pose of the workpiece) in a way that is ergonomically most comfortable for the interacting user. Hence, human-robot interaction is continuously re-evaluated optimizing ergonomics states. The approach is validated through an experimental study, based on established ergonomic methods and their adaptation for real-time application. The study confirms improved ergonomics using the new approach.

Assistive robotic systems endeavour to support those with movement disabilities, enabling them to move again and regain functionality. Main issue with these systems is the complexity of their low-level control, and how to translate this to simpler, higher level commands that are easy and intuitive for a human user to interact with. We have created a multi-modal system, consisting of different sensing, decision making and actuating modalities, leading to intuitive, human-in-the-loop assistive robotics. The system takes its cue from the user's gaze, to decode their intentions and implement low-level motion actions to achieve high-level tasks. This results in the user simply having to look at the objects of interest, for the robotic system to assist them in reaching for those objects, grasping them, and using them to interact with other objects. We present our method for 3D gaze estimation, and grammars-based implementation of sequences of action with the robotic system. The 3D gaze estimation is evaluated with 8 subjects, showing an overall accuracy of 4.68\pm 0.14cm. The full system is tested with 5 subjects, showing successful implementation of 100% of reach to gaze point actions and full implementation of pick and place tasks in 96%, and pick and pour tasks in 76% of cases. Finally we present a discussion on our results and what future work is needed to improve the system.

Robotic systems are ever more capable of automation and fulfilment of complex tasks, particularly with reliance on recent advances in intelligent systems, deep learning and artificial intelligence in general. However, as robots and humans come closer together in their interactions, the matter of interpretability, or explainability of robot decision-making processes for the human grows in importance. A successful interaction and collaboration would only be possible through mutual understanding of underlying representations of the environment and the task at hand. This is currently a challenge in deep learning systems. We present a hierarchical deep reinforcement learning system, consisting of a low-level agent handling the large actions/states space of a robotic system efficiently, by following the directives of a high-level agent which is learning the high-level dynamics of the environment and task. This high-level agent forms a representation of the world and task at hand that is interpretable for a human operator. The method, which we call Dot-to-Dot, is tested on a MuJoCo-based model of the Fetch Robotics Manipulator, as well as a Shadow Hand, to test its performance. Results show efficient learning of complex actions/states spaces by the low-level agent, and an interpretable representation of the task and decision-making process learned by the high-level agent.

Visual context plays a crucial role in understanding human visual attention in natural, unconstrained tasks - the objects we look at during everyday tasks provide an indicator of our ongoing attention. Collection, interpretation, and study of visual behaviour in unconstrained environments therefore is necessary, however presents many challenges, requiring painstaking hand-coding. Here we demonstrate a proof-of-concept system that enables real-time annotation of objects in an egocentric video stream from head-mounted eye-tracking glasses. We concurrently obtain a live stream of user gaze vectors with respect to their own visual field. Even during dynamic, fast-paced interactions, our system was able to recognise all objects in the user's field-of-view with moderate accuracy. To validate our concept, our system was used to annotate an in-lab breakfast scenario in real time.

Humans process high volumes of visual information to perform everyday tasks. In a reaching task, the brain estimates the distance and position of the object of interest, to reach for it. Having a grasp intention in mind, human eye-movements produce specific relevant patterns. Our Gaze-Contingent Intention Decoding Engine uses eye-movement data and gaze-point position to indicate the hidden intention. We detect the object of interest using deep convolution neural networks and estimate its position in a physical space using 3D gaze vectors. Then we trigger the possible actions from an action grammar database to perform an assistive movement of the robotic arm, improving action performance in physically disabled people. This document is a short report to accompany the Gaze-contingent Intention Decoding Engine demonstrator, providing details of the setup used and results obtained.

The low-cost manufacturing and maintenance of prostheses is of vital importance to their successful deployment in developing countries. Low-cost prosthesis actuation is generally achieved by combining pre-programmed control strategies, with surface-electromyographic measurements taken from the residual limb. In a standard setting, these signals are measured with disposable gel electrodes. However, this limit on electrode reuse requires that prosthesis users have a stable supply of electrodes. Alternatively, the textile electrodes sewn from conductive thread are studied in the context of hand gesture recognition to consider their future use with low-cost prostheses. In this paper, it is demonstrated that textile electrodes can be applied for gesture recognition. To do so, surface electromyography (sEMG) experiments are run in South Africa on three amputees where they were asked to perform gestures with their phantom limb (i.e., the missing limb segment). A gesture recognition method is implemented, and the classification accuracy with data recorded from textile electrodes is compared to that from gel electrodes. Further analysis examining the relationship between classifier performance and physiological parameters are performed. Results show that textile electrodes can be used to perform accurate gesture recognition, and are comparable to disposable gel electrodes. This demonstrates that low-cost sensory systems are not barrier to myoelectric control in developing countries.

Since December 2014, FourByThree Project (“Highly customizable robotic solutions for effective and safe human robot collaboration in manufacturing applications”) is developing a new generation of modular industrial robotic solutions that are suitable for efficient task execution in collaboration with humans in a safe way and are easy to use and program by factory workers. This paper summarizes the key technologies that are used to achieve this goal.

Muscle activity monitoring or Electromyography (EMG) is useful in gait analysis, injury prevention, computer or robot interfaces and assisting patients with communication difficulties. However, EMG is typically invasive or obtrusive, expensive and difficult to use for untrained users. A possible solution is textile-based surface EMG (sEMG) integrated into clothing and used as a wearable device. This is, however, challenging due to (i) uncertainties in the electrical properties of conductive threads used to construct electrodes, (ii) imprecise fabrication technologies (e.g., embroidery, sewing), and (iii) a lack of standardisation in the choice of design variables. This paper, for the first time, provides a design guide for such sensors by performing a thorough examination of the effect of design variables on sEMG quality. Electrical characterisation and sEMG measurements are performed, considering the effects of manufacturing imprecision. Results show that the imprecisions in digital embroidery lead to a trade-off between low electrode resistance and high consistency. An optimum set of variables for this trade-off is identified and tested with sEMG during a variable force isometric grip exercise with n=6 participants and compared with traditional gel-based electrodes. Results show that thread-based electrodes provide a similar level of sensitivity to force variation as gel-based electrodes with about 90% correlation with expected linear behaviour.

In this paper, we present a novel pose estimation and obstacle avoidance approach for tendon-driven multi-segment continuum manipulators moving in dynamic environments. A novel multi-stage implementation of an Extended Kalman Filter is used to estimate the pose of every point along the manipulator's body using only the position information of each segment tip. Combined with a potential field, the overall algorithm will guide the manipulator tip to a desired target location and, at the same time, keep the manipulator body safe from collisions with obstacles. The results show that the approach works well in a real-time simulation environment that contains moving obstacles in the vicinity of the manipulator.