Managing indirect access in laparoscopy as a minimally invasive procedure poses challenges to physicians. In particular, an endoscope must be navigated to achieve adequate visualization of the surgical anatomy, while coping with unergonomic poses, tremor, and fatigue. Furthermore, the alignment of visual perception and physical movement, dictated by the endoscope's position relative to the monitor, can lead to hand-eye coordination challenges. We propose unified deployment of a robotic endoscope holder together with an augmented reality display to counteract the aforementioned challenges in laparoscopy. Our augmented reality system provides an interactive, stereoscopic, virtual monitor displaying an endoscopic stream. In addition, our method design enables direct control of the robotic endoscope holder. Our user study demonstrates the potential of the proposed method to significantly improve hand-eye coordination, while insights from our usability study for robotic control indicate promising trends, including high usability and low cognitive demand.

Explainability is vital for establishing user trust, also in robotics. Recently, foundation models (e.g. vision-language models, VLMs) fostered a wave of embodied agents that answer arbitrary queries about their environment and their interactions with it. However, naively prompting VLMs to answer queries based on camera images does not take into account existing robot architectures which represent the robot's tasks, skills, and beliefs about the state of the world. To overcome this limitation, we propose RACCOON, a framework that combines foundation models' responses with a robot's internal knowledge. Inspired by Retrieval-Augmented Generation (RAG), RACCOON selects relevant context, retrieves information from the robot's state, and utilizes it to refine prompts for an LLM to answer questions accurately. This bridges the gap between the model's adaptability and the robot's domain expertise.

Activities like ball bouncing and trampoline jumping showcase the human ability to intuitively tune to system dynamics and excite motions that the system prefers intrinsically. This human sensitivity to resonance has been experimentally supported for interactions with simple linear systems but remains a challenge to validate in more complex scenarios where nonlinear dynamics cannot be predicted analytically. However, it has been found that many nonlinear systems exhibit periodic orbits similar to the eigenmodes of linear systems. These nonlinear normal modes (NNM) are computable with a recently developed numerical mode tool. Using this tool, the present resarch compared the motions that humans excite in nonlinear systems with the predicted NNM of the energy-conservative systems. In a user study consisting of three experiment parts, participants commanded differently configured virtual double pendula with joint compliance through a haptic joystick. The task was to alternately hit two targets, which were either aligned with the NNM (Experiments 1 and 2) or purposefully arranged offset (Experiment 3). In all tested experiment variations, participants intuitively applied a control strategy that excited the resonance and stabilized an orbit close to the ideal NNM of the conservative systems. Even for increased task accuracy (Experiment 2) and targets located away from the NNM (Experiment 3), participants could successfully accomplish the task, likely by adjusting their arm stiffness to alter the system dynamics to better align the resonant motions to the task. Consequently, our experiments extend the existing research on human resonance sensitivity with data-based evidence to nonlinear systems. Our findings emphasize the human capabilities to apply control strategies to excite and exploit resonant motions in dynamic object interactions, including possibly shaping the dynamics through changes in muscle stiffness.

In the present paper, we propose a novel system-driven adaptive shared control framework in which the autonomous system allocates the authority among the human operator and itself. Authority allocation is based on a metric derived from a Bayesian filter, which is being adapted online according to real measurements. In this way, time-varying measurement noise characteristics are incorporated. We present the stability proof for the proposed shared control architecture with adaptive authority allocation, which includes time delay in the communication channel between the operator and the robot. Furthermore, the proposed method is validated through experiments and a user-study evaluation. The obtained results indicate significant improvements in task execution compared with pure teleoperation.

Sending humans to Mars' surface to build habitats is, for now, prohibitively dangerous and costly. An alternative is to have humans in orbiters, teleoperating robots on Mars to construct habitats, deferring human arrival until these habitats are finished. This paper describes the Kontur-2 experiments, in which the feasibility of this scenario was tested with the International Space Station as an orbiter, a cosmonaut operating a force-feedback joystick as an input device for teleoperation, and Earth as the planet where the teleoperated robot is located. In particular, we focus on human teleoperation performance, which is known to deteriorate under conditions of spaceflight. We investigate whether the provision of force feedback at the joystick is as beneficial as under terrestrial conditions. Our results show that, to support humans operating in weightlessness, haptic assistance needs to be adjusted to the altered environmental condition.

This paper presents a new robot controller for space telerobotics missions specially designed to meet the requirements of KONTUR-2, a German & Russian telerobotics mission that addressed scientific and technological questions for future planetary explorations. In KONTUR-2, Earth and ISS have been used as a test-bed to evaluate and demonstrate a new technology for real-time telemanipulation from space. During the August 2015' experiments campaign, a cosmonaut teleoperated a robot manipulator located in Germany, using a force-feedback joystick from the Russian segment of the International Space Station (ISS). The focus of the paper is on the design and performance of the bilateral controller between ISS joystick and Earth robot. The controller is based on a 4-Channels architecture in which stability is guaranteed through passivity and the Time Delay Power Network (TDPN) concept. We show how the proposed approach successfully fulfills mission requirements, specially those related to system operation through space links and internet channels, involving time delays and data losses of different nature.

In the past few years x-by-wire has evolved into the trend-setting technology in the field of transport. It allows for the design of new modular vehicle concepts such as the robotic electric vehicle ROboMObil. Through its four wheel robots, which can be accelerated, decelerated and steered independently, a highly maneuverable design is achieved. A fully actuated three degree of freedom sidestick is proposed in this paper as a haptic interface to the three horizontal degrees of motion of the ROboMObil. The main focus lies on the user access to the rear wheel steering behavior or to the instantaneous center of rotation respectively. Two approaches have been designed and analyzed to improve the robustness against pilot induced oscillations.

A novel concept for visual configuration estimation of a continuum robot is presented, using solely an array of miniature cameras mounted to the surface of the robot as only input sensors. A two-step algorithm is implemented, consisting of a configuration estimation refinement step and a full Bundle Adjustment step, where 3D coordinates and robot configuration parameters are estimated optimally and simultaneously. Real-time capability is provided by fast feature matching procedures and a special solver for the sparse Bundle Adjustment problem. Furthermore, stochastically optimal estimates are guaranteed as the robot parameters are directly estimated and a detour via an individual camera pose estimation is spared. Finally, experimental results are presented to confirm the performance speed and accuracy of the proposed approach.

This work presents an evaluation study of two different collision feedback modalities for virtual assembly verification: visual and force feedback. Forty-three subjects performed several assembly tasks (peg-in-hole, narrow passage) designed with two levels of difficulty. The used haptic rendering algorithm is based on voxel and point data-structures. Both objective - time and collision performance - and subjective measures have been recorded and analyzed. The comparison of the feedback modalities revealed a clear and highly significant superiority of force feedback in virtual assembly scenarios. The objective data shows that whereas the assembly time is similar in most cases for both conditions, force collision feedback yields significantly smaller collision forces, which indicate higher assembly precision. The subjective ratings of the participants define the force feedback condition as the most appropriate for determining clearances and correcting collision configurations, being the best suited modality to predict mountability.

In the present study, a vibrotactile feedback device for spatial guidance was evaluated in a tracking task paradigm. Participants (N = 18) had to translate and rotate virtual objects according to the vibrotactile vs. verbal cues without visual information. Both types of spatial guidance were evaluated using objective performance data (i.e. speed, accuracy) as well as subjective judgments. Results indicate that distinguishing spatial cues during the translational task was more difficult when being guided by vibrotactile feedback compared to verbal feedback. Nevertheless, individuals with vibrotactile guidance showed better performance at rotational tasks. Implications for the further design process and other areas of application are discussed.

In this paper a new approach for robot control using position based visual servoing (PBVS) with an omnidirectional multi-camera system is presented. PBVS requires the explicit calculation of the position and orientation of the robot tool. Given only images without depth information, either additional geometric properties of the observed scene or stereo correspondences have to be provided for pose estimation. In this paper we use triangulation of images given by many cameras pointing in various directions. We confirm this approach with some simulational results.