As space agencies prepare for long-term lunar missions (ie. Lunar Base Camp), addressing the challenges of complex surface tasks is crucial. Integrating advanced robotic systems can alleviate physical and cognitive burdens on astronauts, enhancing mission success and safety. This work focuses on the design, development, and evaluation REACH, a handheld device and its UI for interacting with the modular DLR Lightweight Rover Unit 2 (LRU), to improve human-robot collaboration during lunar EVAs. A Human-In-The-Loop pilot study compares the effectiveness of the REACH device to traditional verbal commands within a modular robotic system. Our results show REACH improves user control, task efficiency, and satisfaction with its intuitive interface and real-time feedback. These findings suggest REACH, and other like systems, can streamline tasks operations and management, aligning with Artemis mission goals. This work is a crucial step toward optimizing Human-Robot Interaction, setting the stage for more effective space missions.

This paper proposes a knowledge-driven teleoperation framework that enables multiple operators to command a team of robots to execute complex tasks in an efficient and intuitive manner. The framework leverages a shared knowledge base that captures domain-specific information and procedural knowledge about the task at hand. This knowledge base is used by a hybrid planner to generate context-specifically relevant commands for supervised autonomy robot command as well as direct teleoperation modes. By filtering the available commands, the operators are guided in their decision-making towards efficient task completion. This paper further extends our knowledge driven approach to address the switching between multiple operators and robotic assets, with the aim to be able scale up human-robot team for space exploration. Overall, this work represents a step towards more intelligent and collaborative teleoperation systems. The described system will be used in the Surface Avatar ISS-to-ground experiments slated for July 2023.

Telerobotic systems must adapt to new environmental conditions and deal with high uncertainty caused by long-time delays. As one of the best alternatives to human-level intelligence, Reinforcement Learning (RL) may offer a solution to cope with these issues. This paper proposes to integrate RL with the Model Mediated Teleoperation (MMT) concept. The teleoperator interacts with a simulated virtual environment, which provides instant feedback. Whereas feedback from the real environment is delayed, feedback from the model is instantaneous, leading to high transparency. The MMT is realized in combination with an intelligent system with two layers. The first layer utilizes Dynamic Movement Primitives (DMP) which accounts for certain changes in the avatar environment. And, the second layer addresses the problems caused by uncertainty in the model using RL methods. Augmented reality was also provided to fuse the avatar device and virtual environment models for the teleoperator. Implemented on DLR's Exodex Adam hand-arm haptic exoskeleton, the results show RL methods are able to find different solutions when changes are applied to the object position after the demonstration. The results also show DMPs to be effective at adapting to new conditions where there is no uncertainty involved.

Advances in exoskeleton technology now enable interacting with rigid objects in a virtual or remote environment using one's hand and fingertips. However, interaction with non-solid materials - such as liquids, sediments and regolith - alongside solids, can greatly extend the versatility of this technology. Rendering rigid objects adequately requires a control loop with high update rates, whereas fluid dynamics equations are computationally expensive. To accommodate this, the fluid dynamics can be simplified - particularly for fluids with high viscosity - resulting in a fast-to-calculate model to enabling haptic rendering of viscous fluids and rigid bodies simultaneously using DLR's Exodex Adam hand exoskeleton. Viscosity as a proprioceptive cue of fluids can be presented to the human through force feedback at multiple points on the human hand - fingers and palm - letting the user interact with a virtual environment in a more natural way and making the experience more immersive. We carry out two user studies to investigate the human perception abilities of virtual fluids rendered with simplified dynamics, and the discernability of different viscosity in virtual fluids compared real fluids. Results show that virtual media can give the user the perception of interacting with a fluid, even with simplified models, at a high update frequency. Furthermore, the material discernibility corresponds well to actual interaction with real viscous fluids. This shows great promise forward for haptic in-hand interaction in fluid and mixed media environments.

Intelligent robotic coworkers are considered a valuable addition in many application areas. This applies not only to terrestrial domains, but also to the exploration of our solar system. As humankind moves toward an ever increasing presence in space, infrastructure has to be constructed and maintained on distant planets such as Mars. AI-enabled robots will play a major role in this scenario. The space agencies envisage robotic co-workers to be deployed to set-up habitats, energy, and return vessels for future human scientists. By leveraging AI planning methods, this vision has already become one step closer to reality. In the METERON SUPVIS Justin experiment, the intelligent robotic coworker Rollin’ Justin was controlled from Astronauts aboard the International Space Station (ISS) in order to maintain a Martian mock-up solar panel farm located on Earth to demonstrate the technology readiness of the developed methods. For this work, the system is demonstrated at AAAI 2019, controlling Rollin’ Justin located in Munich, Germany from Honolulu, Hawaii.

Accurate real-time estimation of the pose and configuration of the human hand attached to a dexterous haptic input device is crucial to improve the interaction possibilities for teleoperation and in virtual and augmented reality. In this paper, we present an approach to reconstruct the pose of the human hand and the joint angles of the fingers when wearing a novel fixed-base (grounded) hand exoskeleton. Using a kinematic model of the human hand built from MRI data, we can reconstruct the hand pose and joint angles without sensors on the human hand, from attachment points on the first three fingers and the palm. We test the accuracy of our approach using motion capture as a ground truth. This reconstruction can be used to determine contact geometry and force-feedback from virtual or remote objects in virtual reality or teleoperation.

This paper presents an adaptive and compliant approach-to-grasp strategy for multi-finger robotic hands, to improve the performance of autonomous grasping when encountering object position uncertainties. With the proposed approach-to-grasp strategy, the first robot finger to experience unexpected impact would pause its movement in a compliant manner, and remains in contact with the object to minimize the unplanned motion of the target object. At the same time, the remainder of the fingers continuously, adaptively move toward re-adjusted grasping positions with respect to the first finger in contact with the object, without the need for on-line re-planning or re-grasping. An adaptive grasp control strategy based on spatial virtual spring framework is proposed to achieve local (e.g. not resorting to the robotic arm) in-hand adjustments of the fingers not yet in contact. As such, these fingers can be adaptively driven to the adjusted desired position to accomplish the grasp. Experimental results demonstrate that significantly larger position errors with respect to the hand workspace can be accommodated with the proposed adaptive compliant grasp control strategy. As much as 391% increase in position error area coverage has been achieved. Finally, beyond the quantitative analysis, additional observations during the extensive experiment trials are discussed qualitatively, to help examine several open issues, and further understand the approach-to-grasp phases of the robot hand tasks.

Dexterous multifingered hands are the most complex and versatile variants of robotic end effectors. Compared to simpler grippers and underactuated hands, they should be more capable of grasping and, especially, manipulating different objects. This paper explores the relationship between kinematic design and manipulation performance of robotic hands. Some evaluation criteria frequently used by hand designers to verify kinematic configurations are revisited. The results from these criteria are scrutinized and compared with the evaluation of the manipulation workspace and the ranges of motion of inhand manipulation for a set of predefined objects. Simulations and actual manipulation experiments are carried out with different kinematic configurations on a modular dexterous hand. The results show some disconnection between perceived good designs through common evaluation criteria and their actual, realizable manipulation performance. This work finally gives some insight toward a more holistic approach to design hands that better address grasp and manipulation for the intended tasks and applications.

This paper introduces a new framework for task space telemanipulation. The TASK space grasping and MANipulation (TaskMan) concept utilizes a library of tasks based on gesture commands, which replaces the conventional mapping required between the human hand and the end effector. Task communication between the human machine interface (HMI) and the robot end effector requires two symbiotic but nonidentical state machines on the master and slave side. The task states on two sides are synchronized via a single channel communication, as opposed to multi-channel joint space or Cartesian mapped information. HAND gesture command for grasping and MANipulation (HandyMan) HMI command algorithm is proposed for the recognition of hand gestures, which incorporates a library of intuitive task gestures to be used by the teleoperator wearing a CyberGlove. The task gestures are used to drive the states of the TaskMan state machines. With the proposed concepts, this work has realized teleoperated grasp and manipulation with a 15-DoF robot hand in task space. Full 6-DoF of object manipulation was achieved with different grasp combinations, and demonstrated higher repeatability, success rate and easier operation compared to conventional joint space teleoperation methods.

This paper presents experimental results on the five-finger dexterous robot hand DLR-HIT II, with Cartesian impedance control based on joint torque and nonlinearity compensation for elastic dexterous robot joints. To improve the performance of the impedance controller, system parameter estimations with extended kalman filter and gravity compensation have been investigated on the robot hand. Experimental results show that, for the harmonic drive robot hand with joint toruqe feedback, accurate position tracking and stable torque/force response can be achieved with cartesian and joint impedance controller. In addition, a FPGA-based control architecture with flexible communication is proposed to perform the designed impedance controller.

This paper introduces a study aimed to help quantify the benefits of limited-performance force-feedback user input devices for space telemanipulation with a dexterous robotic arm. A teleoperated robotic hand has been developed for the European Space Agency by the German Aerospace Center (DLR) for a lunar rover prototype. Studies carried out on this telerobotic system investigated several criteria critical to telemanipulation in space: (1) grasping task completion time, (2) grasping task difficulty, (3) grasp quality, and (4) difficulty level for the operator to assess the grasp quality. Several test subjects were allocated to remotely grasp regular and irregular shaped objects, under different combinations of visual- and force-feedback conditions. This work categorized the benefits of visual- and force-feedback in teleoperated grasping through several performance metrics. Furthermore, it has been shown that, with local joint-level impedance control, good grasping performance with rigid hard objects can be achieved, even with limited force-feedback information and low communication bandwidth. On the other hand, a performance ceiling was also found when grasping deformable objects, where the limited force-feedback setup cannot sufficiently reflect the object boundary to the teleoperator.

Dissimilar sensor redundancy with force and displacement measurements can provide extended fault coverage for drive-by-wire applications. However, large variances occur when correlating these inputs in dynamic measurements. To address this issue, this paper introduces a novel fault tolerant sensor fusion concept to fuse redundant kinetic and kinematic sensors dynamically and accurately. It consists of a driver input condition classifier, and a graceful degradation scheme voter. The classifier determines the states of several driver input conditions in real-time, which are then utilized as a pointers to allocate the best fitting driver input kinetic-kinematic response profile during dynamic operation for sensor data conversion. The graceful degradation voter module facilitates voting and fusing of the converted redundant sensor data. Functional safety is further maintained through a graceful degradation scheme to tolerate the presence of different fault scenarios in the system. The proposed sensor fusion concept is implemented on a brake-by-wire pedal test bed. Test results show significant performance gains in driver command accuracy for fault-tolerant dissimilar redundant sensor fusion.