In this paper, we propose a concept of Teleoperated Teaching of Task and Impedance (TTTI) with a novel multi-modal interface that enables online teleoperated teaching of combined low-level impedance-regulation skills and high-level task decision-making skills using a single hand-held haptic device. To this end, we interactively switch the functionality of the haptic device for two modes of operation. To teach impedance-regulation low-level skills, we developed a novel stiffness command interface where the human operator uses the haptic device to manipulate the stiffness ellipsoid of the remote robotic arm endpoint in 3D space. For teaching high-level skills of how and when to employ low-level actions, we developed a GUI that enables a haptic device to remotely modify Behaviour Trees used to encode the robot’s task decision-making process. The interface connects both teaching modes, where a newly demonstrated low-level skill appears in the Behaviour Tree at an operator-specified index. To demonstrate the main features of the proposed interface, we performed several proof-of-concept experiments on a teleoperation setup operating a remote shelf-stocker robot in a simulated supermarket environment. We examined the task of placing a product on a shelf that consists of several sub-tasks, where each involves different stiffness strategies, while the Behaviour Tree has to encode the task sequencing and decision-making process.

In this work, we propose a method of capturing the patient’s discomfort during robotic shoulder physiotherapy, creating "discomfort maps". These maps depict the personalized distribution of discomfort that each patient perceived across their shoulder range of motion, facilitating both robotic devices and human therapists to account for patient-specific characteristics during the therapeutic process. Our system enables a patient to communicate and map discomfort in space and time during movement via a handheld push-button device, while interacting with a robotic physical therapy device capable of moving the patient and estimating their pose. We validated our method through human factors experiments simulating shoulder physiotherapy sessions with 10 healthy participants. To avoid the risk of injury to the participants and to allow for ground truth map information, we emulate perceived discomfort via an auditory signal. Our experimental apparatus enabled participants to reconstruct synthetic discomfort maps, demonstrating the feasibility of automatically capturing and storing patient discomfort during robotic physiotherapy.

Teleoperation is a crucial technology enabling human operators to control robots remotely to perform tasks in hazardous and difficult-to-access environments. Tasks in such environments often involve complex physical interactions with tools and objects of various softness. To this end, teleimpedance enables the operators to adjust the robot impedance in real-time to simplify such interactions. While the existing teleimpedance approaches provide several interfaces to command the robot impedance, there are no interfaces to visualize both the commanded impedance and that of the objects to be interacted with. This paper presents a novel interface to provide visual feedback on the impedance of remote robots and objects. To do so, we use virtual stiffness ellipsoids and different modes that display the individual impedance of the robot and objects as well as combined post-contact impedance. The key advantage of visual feedback on the impedance compared to force feedback is that the operator can see the interaction characteristics before the contact occurs. This enables the operator to act proactively before contact rather than just reactively after the contact. This paper also proposes a new intuitive way to command the robot impedance using mixed reality, interacting with these ellipsoids and modifying them as needed. To demonstrate the key functionalities of the developed interface, we performed proof-of-concept experiments on teleoperated tasks.

Robotic teleoperation is used in various applications, including the nuclear industry, where the experience and intelligence of a human operator are necessary for making complex decisions that are beyond the autonomy of robots. Human-robot interfaces that help strengthen an operators situational awareness without inducing excessive cognitive load are crucial to the success of teleoperation. This paper presents a novel visual interface that allows operators to simultaneously control a 6-DoF camera platform and a robotic manipulator whilst experiencing the remote environment through a virtual reality head-mounted display (HMD). The proposed system, Orbital Head-Mounted Display (OHMD), utilizes head rotation tracking to command camera movement in azimuth and elevation directions around a fixation point located at a robot's end-effector. A human factor study was conducted to compare the interface acceptance, perceived workload, and task performance of OHMD with a conventional interface utilizing multiple fixed cameras (Array) and a standard head-mounted display implementation (HMD). Results show that both the OHMD and HMD interfaces significantly improve task performance, reduce perceived workload and increase interface acceptance compared to the Array interface. Participants reported they preferred OHMD due to the increased assistance and freedom in viewpoint selection. Whilst OHMD excelled in usefulness, the standard HMD interface allowed operators to perform robotic welding tasks significantly faster.

Despite the significant progress made in making robots more intelligent and autonomous, today, teleoperation remains a dominant robot control paradigm for the execution of complex and highly unpredictable tasks. Attempts have been made to make teleoperation systems stable, easy to use, and efficient in terms of physical interactions between the follower remote robot and the environment. In particular, the emergence of torque-controlled robots has permitted to regulate the interaction forces from a distance through direct force or impedance control, enabling them to engage in complex interaction tasks. Exploiting this feature, the concept of teleimpedance control was introduced as an alternative method to bilateral force-reflecting teleoperation. The aim was to create a feed-froward yet contact-efficient teleoperation by enriching the leader commands with desired impedance profiles while executing a task. Since then, the teleimpedance concept has found its way into a wide range of interface and controller designs, as well as application domains. Accordingly, after a decade of research progress, this survey aims to provide: first, a convenient introduction of the concept to new researchers in the field, second, consolidate the existing state-of-the-art for active researchers, third, and discuss the pros and cons of different methods in terms of interface and force feedback to provide guidelines for different applications and future developments.

Physical human-robot interaction can improve human ergonomics, task efficiency, and the flexibility of automation, but often requires application-specific methods to detect human state and determine robot response. At the same time, many potential human-robot interaction tasks involve discrete modes, such as phases of a task or multiple possible goals, where each mode has a distinct objective and human behavior. In this paper, we propose a novel method for multi-modal physical human-robot interaction that builds a Gaussian process model for human force in each mode of a collaborative task. These models are then used for Bayesian inference of the mode, and to determine robot reactions through model predictive control. This approach enables optimization of robot trajectory based on the belief of human intent, while considering robot impedance and human joint configuration, according to ergonomic- and/or task-related objectives. The proposed method reduces programming time and complexity, requiring only a low number of demonstrations (here, three per mode) and a mode-specific objective function to commission a flexible online human-robot collaboration task. We validate the method with experiments on an admittance-controlled robot, performing a collaborative assembly task with two modes where assistance is provided in full six degrees of freedom. It is shown that the developed algorithm robustly re-plans to changes in intent or robot initial position, achieving online control at 15 Hz.

Tele-impedance increases interaction performance between a robotic tool and unstructured/unpredictable en-vironments during teleoperation. However, the existing tele-impedance interfaces have several ongoing issues, such as long calibration times and various obstructions for the human operator. In addition, they are all designed to be controlled by the operator's arms, which can cause difficulties when both arms are used, as in bi-manual teleoperation. To resolve these issues, we designed a novel foot-based tele-impedance control method inspired by the human limb stiffness ellipse modulation. The proposed mechanical interface design includes a disc and a foot pressure sensor that controls the orientation and size/shape of the stiffness ellipse, respectively. We evaluated the disc interface control method in an experimental study with 12 participants, who performed a complex drilling task in a virtual environment. The results show the ability of the operator to use the proposed interface in order to dynamically adapt to different phases of the task and changes in the environment. In addition, a comparison with low and high uniform impedance modes demonstrates a superior interaction performance of the proposed method.

Teaching robots how to apply forces according to our preferences is still an open challenge that has to be tackled from multiple engineering perspectives. This paper studies how to learn variable impedance policies where both the Cartesian stiffness and the attractor can be learned from human demonstrations and corrections with a user-friendly interface. The presented framework, named ILoSA, uses Gaussian Processes for policy learning, identifying regions of uncertainty and allowing interactive corrections, stiffness modulation and active disturbance rejection. The experimental evaluation of the framework is carried out on a Franka-Emika Panda in four separate cases with unique force interaction properties: 1) pulling a plug wherein a sudden force discontinuity occurs upon successful removal of the plug, 2) pushing a box where a sustained force is required to keep the robot in motion, 3) wiping a whiteboard in which the force is applied perpendicular to the direction of movement, and 4) inserting a plug to verify the usability for precision-critical tasks in an experimental validation performed with non-expert users.

Tele-impedance augments classic teleoperation by enabling the human operator to actively command remote robot stiffness in real-time, which is an essential ability to successfully interact with the unstructured and unpredictable environment. However, the literature is missing a study on benefits and drawbacks of different types of stiffness command interfaces used in bilateral tele-impedance. In this article, we introduce a term called coupling effect, which pertains to the coupling between human-commanded stiffness going to the remote robot and force feedback coming from the remote robot. We hypothesize that, whenever the operator's commanded stiffness and force feedback are subject to coupling effect (e.g., muscle activity based stiffness command interfaces), force feedback can invoke involuntary changes in the commanded stiffness due to human reflexes. Although the coupling effect takes away some degree of the operator's control over the commanded stiffness, these involuntary changes can be either beneficial (e.g., during position tracking) or detrimental (e.g., during force tracking) to the task performance on the remote robot side. We examined the coupling effect in an experimental study with16participants, who performed position and force tracking tasks by using both coupled type (muscle activity based) and decoupled type (external device based) of interface. The results demonstrate a benefit of the coupling effect when the remote robot is operating in presence of unexpected force perturbations, where lower absolute error in position tracking task was observed. On the other hand, the decoupled type of interface is beneficial for force tracking tasks on the remote robot side, such as establishing or maintaining a stable contact with objects. However, the coupling effect negatively influences the commanding of reference stiffness to the remote robot in both position and force tracking tasks for the coupled type of interface, compared to the decoupled type of interface, which is not affected.

In this paper, we propose a novel method for selective management of muscle fatigue in human-robot co-manipulation. The proposed framework enables the detection of excessive fatigue levels of an individual muscle group while executing a certain task, and provides anticipatory robotic responses to distribute the effort among less-fatigued muscles of human arm. Our approach uses a machine learning technique to enable online predictions of muscle forces in different arm configurations and endpoint interaction forces. The estimated muscle forces are then used for the model-based estimation of muscle fatigue levels. Through optimisation, the fatigue management system can alter the task execution in a way that specific fatigued muscles are offloaded, while at the same time enables the production of task force using muscles with lower levels of fatigue. The main advantage of the proposed method is that it can operate online, and that all the measurements are performed by the robot sensory system, which can significantly increase the applicability in real-world scenarios. To validate the proposed method, we performed proof-of-concept experiments where the task of the human operator was to use a tool to polish an object that was manipulated by the robot.

Collaborative robots are often designed with limited power and force capacity, with the aim to provide affordable solutions and ensure human safety in case of accidental collisions and impacts. If a task requires a power beyond this capacity, or is performed repeatedly over long periods, such limits may be exceeded, which can cause inevitable robot damage and contribute to the lost productivity. In such cases, where hardware solutions and improvements are not applicable, effective software frameworks can prolong robot productivity and lifetime. To this end, in this paper we propose a novel technique for the monitoring and management of robot fatigue in repetitive or high-effort task execution scenarios. The robot fatigue is estimated by the measured temperature of motors in the joints. The proposed fatigue management system is composed of two-stage reaction process that is triggered by different levels of the estimated fatigue. The first stage exploits the kinematic redundancy of robot structure in attempt to minimise the load in the specific joints that under fatigue by reconfiguration in the joint space through the null space of the Cartesian task production. If the first stage is not successful in reducing the fatigue, the second stage is activated that gradually reduces the forces of hybrid controller. At that point, the human co-worker can temporarily take over the task execution until the robot will be recovered from the excessive fatigue. To validate the proposed approach we conducted experiments on KUKA Lightweight Robot performing two interaction tasks: autonomous surface wiping and collaborative human-robot surface polishing.

The paper presents a novel control method for the arm exoskeletons that takes into account the muscular force manipulability of the human arm. In contrast to classical controllers that provide assistance without considering the biomechanical properties of the human arm, we propose a control method that takes into account the configuration of the arm and the direction of the motion to effectively compensate the anisotropic property of the muscular manipulability of the human arm. Consequently, the proposed control method effectively maintains a spherical endpoint manipulability in the entire workspace of the arm. As a result, the proposed method allows the human using the exoskeleton to efficiently perform tasks in arm configurations that are normally unsuitable due to the low manipulability. We evaluated the proposed approach by a preliminary experimental study where a subject wearing a 2 DOF arm-exoskeleton had to move a 4 kg weight between several locations. The results of our study demonstrate that the proposed approach effectively augments the ability of human motor control to perform tasks equally well in the whole arm workspace that include configurations with low intrinsic manipulability.

We propose a novel method that arbitrates the control between the human and the robot actors in a teaching-by-demonstration setting to form synergy between the two and facilitate effective skill synthesis on the robot. We employed the human-in-the-loop teaching paradigm to teleoperate and demonstrate a complex task execution to the robot in real-time. As the human guides the robot to perform the task, the robot obtains the skill online during the demonstration. To encode the robotic skill we employed Locally Weighted Regression that fits local models to specific state region of the task based on the human demonstration. If the robot is in the state region where no local models exist, the control over the robotic mechanism is given to the human to perform the teaching. When local models are gradually obtained in that region, the control is given to the robot so that the human can examine its performance already during the demonstration stage, and take actions accordingly. This enables a co-adaptation between the agents and contributes to a faster and more efficient teaching. As a proof-of-concept, we realised the proposed robot teaching system on a haptic robot with the task of generation of a desired vertical force on a horizontal plane with unknown stiffness properties.

This paper presents a novel approach for human-robot cooperation in tasks with dynamic uncertainties. The essential element of the proposed method is a multi-modal interface that provides the robot with the feedback about the human motor behaviour in real-time. The human muscle activity measurements and the arm force manipulability properties encode the information about the motion and impedance, and the intended configuration of the task frame, respectively. Through this human-in-the-loop framework, the developed hybrid controller of the robot can adapt its actions to provide the desired motion and impedance regulation in different phases of the cooperative task. We experimentally evaluate the proposed approach in a two-person sawing task that requires an appropriate complementary behaviour from the two agents.

In this paper we propose a human-in-the-loop approach for teaching robots how to solve part assembly tasks. In the proposed setup the human tutor controls the robot through a haptic interface and a hand-held impedance control interface. The impedance control interface is based on a linear spring-return potentiometer that maps the button position to the robot arm stiffness. This setup allows the tutor to modulate the robot compliance based on the given task requirements. The demonstrated motion and stiffness trajectories are encoded using Dynamical Movement Primitives and learnt using Locally Weight Regression. To validate the proposed approach we performed experiments using Kuka Light Weight Robot and HapticMaster robot. The task of the experiment was to teach the robot how to perform an assembly task involving sliding a bolt fitting inside a groove in order to mount two parts together. Different stiffness was required in different stages of the task execution to accommodate the interaction of the robot with the environment and possible human-robot cooperation.

In this paper we propose a system capable of teaching humanoid robots new skills in real-time. The system aims to simplify the robot control and to provide a natural and intuitive interaction between the human and the robot. The key element of the system is exploitation of the human sensorimotor learning ability where a human demonstrator learns how to operate a robot in the same fashion as humans adapt to various everyday tasks. Another key aspect of the proposed system is that the robot learns the task simultaneously while the human is operating the robot. This enables the control of the robot to be gradually transferred from the human to the robot during the demonstration. The control is transferred based on the accuracy of the imitated task. We demonstrated our approach using an experiment where a human demonstrator taught a humanoid robot how to maintain the postural stability in the presence of the perturbations. To provide the appropriate feedback information of the robot's postural stability to the human sensorimotor system, we utilized a custom-built haptic interface. To absorb the demonstrated skill by the robot, we used Locally Weighted Projection Regression machine learning method. A novel approach was implemented to gradually transfer the control responsibility from the human to the incrementally built autonomous robot controller.