Teleoperation with haptic feedback allows users to interact with remote environments while retaining a sense of touch. However, the stability and transparency of these systems are compromised under communication network delay. This paper presents an augmented Model-Mediated Teleoperation with 3D object and dynamic environment tracking (MMT-DET) by a vision-based algorithm, enabling users to receive haptic feedback in structured dynamic environments while maintaining robustness against network delays. A user study comparing the proposed method with teleoperation using the Time Domain Passivity Approach (TDPA) was conducted. The results demonstrate that our MMT-DET exhibits robustness to varying delays in task performance and outperforms TDPA at higher delay levels.

Integrating humanoid service mobile robots into human environments presents numerous challenges, primarily concerning the safety of interactions between robots and humans. To address these safety concerns, we propose a novel approach that leverages the capabilities of digital twin technology by tailoring it to incorporate comprehensive and robust safety concepts. This paper introduces a "safe-by-design" digital twin that operates alongside the real twin robot in the loop, engaging real-time safety framework during physical interactions with the surrounding environment, including humans.To validate the effectiveness of our proposed safe-by-design digital twin framework, we conducted experiments using a humanoid service mobile robot alongside simulated human counterparts. Our results demonstrate the capability of the integrated impact safety module within the proposed digital twin approach to limit the velocities of both the robot’s base and arms, adhering to injury biomechanics-based safety thresholds. These findings emphasize the promise of our proposed approach for ensuring the physical safety of humanoid service mobile robots operating in dynamic human environments. It enables the digital twin to preemptively identify potential safety hazards and formulate safe intervention actions to ensure the robot’s compliance with safety regulations, paving the way for safer and more widespread adoption of robotic systems in various service domains.

The need for comprehensive telemedicine solutions is becoming increasingly relevant due to challenges associated with the ageing population, the increasing shortage of health-care providers, and, more recently, the global pandemic. Existing solutions primarily focus on, e.g., electronic medical records, audiovisual connections, and, in some cases, robotic systems with very basic capabilities. Here we present a fundamentally new, holistic approach to a remote doctor visit, which enables transparent remote examination, anomaly detection, diagnosis, and rehabilitation. Our dual doctor-patient twin paradigm involves two robotic systems: one representing the doctor to the patient ("GARMI") and one representing the patient to the doctor ("MUCKI"). Through bidirectional telepresence control, this system enables transparent, natural, remote haptic interaction between doctor and patient. The control, interaction, and knowledge transfer to the doctor is enhanced by AI-based visual motion and facial expression analysis as well as a digital twin of the patient. Thus, each stage of a doctor visit can be replicated in the context of telemedicine and shared autonomy: from first assessment to observation-based and remote physical examination, to a better-informed doctor diagnosis and robot-assisted telerehabilitation.