Ensuring the sterility of medical equipment, particularly endoscopes used in environments teeming with diverse pathogens and drug-resistant bacteria, is crucial for safe medical procedures. However, the complexity of endoscope reprocessing, which involves numerous dexterous manual manipulations, poses significant challenges. Achieving certification for sterilization requires precise, repetitive execution with strict tolerances. In this study, we propose a framework that automates the handling and storage of endoscopes right after the sterilization process and employs compliant collaborative robots to address these dexterous manipulation challenges. In the first stage, we identified the key manipulation skills involved in the process through observations and feedback from medical personnel. In the second stage, we proposed a system that employs a high-level action planner to orchestrate the removal and storage of endoscopes, integrating two collaborative robots and a linear unit. Through real-time force measurements, compliant control, task knowledge, and safety protocols, we establish a system that ensures the safety of both medical equipment and personnel in proximity. In our first experiment, we conducted 50 trials with a 100 % reliability rate. Each trial had an execution time of 102 seconds, with a variance of 1.2 seconds. In our second experiment, we performed 10 trials with a human obstructing the transfer path, facing away from the robot. In all cases, the system successfully and promptly detected the collision. This work pioneers the automation of medical reprocessing in sterile environments using tactile robots and addresses the associated challenges.

This paper presents a novel approach for robot navigation in environments containing deformable obstacles. By integrating Learning from Demonstration (LfD) with Dynamical Systems (DS), we enable adaptive and efficient navigation in complex environments where obstacles consist of both soft and hard regions. We introduce a dynamic modulation matrix within the DS framework, allowing the system to distinguish between traversable soft regions and impassable hard areas in real-time, ensuring safe and flexible trajectory planning. We validate our method through extensive simulations and robot experiments, demonstrating its ability to navigate deformable environments. Additionally, the approach provides control over both trajectory and velocity when interacting with deformable objects, including at intersections, while maintaining adherence to the original DS trajectory and dynamically adapting to obstacles for smooth and reliable navigation.

In this paper, we extend the Unified Force-Impedance Control (UFIC) framework for the whole-body control of service mobile robots subject to non-holonomic constraints. This enables the robot to execute complex service tasks that demand both force and impedance control within its whole body workspace. The task space of the robot is defined as the pose of both end-effectors. Following the concept of UFIC, both impedance and force tracking commands are applied in the task space of the whole-body controller, with augmented energy tanks incorporated to ensure passivity. To enable smooth transitions between force tracking and impedance control—particularly in cases of contact loss—a shaping function is used to modulate the force control command. Additionally, the robot’s redundancy is exploited to shape the posture, while satisfying joint limits, avoiding singularities, and preventing self-collisions between the arms. The effectiveness of the proposed whole-body UFIC controller is validated through simulations and real-world experiments with the service robot GARMI performing several daily tasks.

Teleoperation with haptic feedback allows users to interact with remote environments while retaining a sense of touch. However, the stability and transparency of these systems are compromised under communication network delay. This paper presents an augmented Model-Mediated Teleoperation with 3D object and dynamic environment tracking (MMT-DET) by a vision-based algorithm, enabling users to receive haptic feedback in structured dynamic environments while maintaining robustness against network delays. A user study comparing the proposed method with teleoperation using the Time Domain Passivity Approach (TDPA) was conducted. The results demonstrate that our MMT-DET exhibits robustness to varying delays in task performance and outperforms TDPA at higher delay levels.

This paper presents GARMI’s successful outdoor demonstration during the Kandahar Ski World Cup, where it performed trophy handovers in sub-zero temperatures. The event highlighted challenges in deploying robots in extreme conditions, including fluctuating temperatures and uneven terrain. GARMI achieved and completed the trophy handover during the live event, streamed to 60 million viewers. This experience raised two key research questions: the feasibility of high-precision robotics in harsh weather and strategies to compensate for environmental effects. To address them, we extended our previous framework to estimate the mass of the lifted trophy in real-time, incorporating IMU data and conducting experiments under varying temperatures and orientations. Experimental results showed that even slight variations in the robot’s base orientation had a significant impact on the accuracy of mass estimation. For instance, a 5° tilt in the robot’s base orientation resulted in a more than 100% increase in mass estimation error. Online mass estimation, performed using a quasi-static model, demonstrated improved accuracy when incorporating IMU-based corrections for base orientation. Additionally, temperature variations were found to affect robot control performance, with tracking errors increasing outside the manufacturer’s recommended temperature range. The findings highlighted the need for real-time corrections and compensations for base orientation and temperature in robot dynamics, ensuring safe human-robot interaction.

In this work, we advance robotic grasping by incorporating wrist compliance in a unified hand-arm system inspired by human limb coordination. This integration improves grasping reliability and robustness through impedance and force learning in robotic arms. The compliant wrist system effectively compensates for uncertainties in object position and orientation. Employing a combined impedance-force control approach, we address diverse grasping and manipulation tasks in simulation. Successfully transferring the learned policy to a service humanoid mobile robot enables the seamless execution of grasping and opening tasks for various doors and handles without additional learning, using both fully actuated and underactuated robotic hands. Remarkably, our robust strategies yielded only one failure in 30 trials for the underactuated hand, even with up to 8 cm translation normal to the handle and$33^\circ$rotation errors, and no failures for the fully actuated one with up to 12 cm translation and$30^\circ$rotation. This significantly outperforms state-of-the-art end-to-end reinforcement learning approaches. Furthermore, we successfully tested and validated our approach across various constrained everyday tasks in different environments. Our proposed framework represents an advancement in the learning and execution of power grasping with compliant manipulation, achieving practically relevant performance.

A drawing robot avatar is a robotic system that allows for telepresence-based drawing, enabling users to remotely control a robotic arm and create drawings in real-time from a remote location. The proposed control framework aims to improve bimanual robot telepresence quality by reducing the user workload and required prior knowledge through the automation of secondary or auxiliary tasks. The introduced novel method calculates the near-optimal Cartesian end-effector pose in terms of visual feedback quality for the attached eye-to-hand camera with motion constraints in consideration. The effectiveness is demonstrated by conducting user studies of drawing reference shapes using the implemented robot avatar compared to stationary and teleoperated camera pose conditions. Our results demonstrate that the proposed control framework offers improved visual feedback quality and drawing performance.

This paper investigates the problem of effective tool manipulation for motion planning in complex human-like scenarios. Vector-field-based real-time strategies, although widely used, usually do not account for unwieldy tools or incorporate systematic methods to handle these extra maneuvers needed. Instead, we formalize the problem and propose a novel field-based reactive planner that explicitly accounts for rotational forces for seamless maneuvers based on the tool’s geometry and featured points. Furthermore, we capture and encode robot performance through capability metrics and improve the same using an additional quality distribution method. This enables seamless integration of the robot’s embodiment with the reactive force-torque (wrench) field giving rise to flexible tool usage in non-stationary environments. Extensive simulation analysis on a 7 DoF collaborative robot manipulating a common tool in an unorganized table-top layout reinforces our claim of robustness in stationary and non-stationary scenarios.

Bowden cables serve as essential components in various mechanical systems, facilitating power transmission from remote actuators to specific destinations. The pretension of Bowden cables profoundly influences system performance, notably in terms of friction. This study investigates the effects of cable pretension and shape on friction and torque efficiency. A custom self-designed testbed, comprising integrated actuator units, pulleys, and a novel pretension mechanism connected by Bowden cables, is utilized to conduct experimental tests under varying parameters. This work adopts an integrated approach of experimentation, modeling, and validation, offering preliminary insights into the torque transmission characteristics of tendon driven actuator systems. Additionally, the precise model exhibits excellent conformity across a broad range of shapes and provides initial insights into hysteresis modeling attributable to cable material properties.

Integrating humanoid service mobile robots into human environments presents numerous challenges, primarily concerning the safety of interactions between robots and humans. To address these safety concerns, we propose a novel approach that leverages the capabilities of digital twin technology by tailoring it to incorporate comprehensive and robust safety concepts. This paper introduces a "safe-by-design" digital twin that operates alongside the real twin robot in the loop, engaging real-time safety framework during physical interactions with the surrounding environment, including humans.To validate the effectiveness of our proposed safe-by-design digital twin framework, we conducted experiments using a humanoid service mobile robot alongside simulated human counterparts. Our results demonstrate the capability of the integrated impact safety module within the proposed digital twin approach to limit the velocities of both the robot’s base and arms, adhering to injury biomechanics-based safety thresholds. These findings emphasize the promise of our proposed approach for ensuring the physical safety of humanoid service mobile robots operating in dynamic human environments. It enables the digital twin to preemptively identify potential safety hazards and formulate safe intervention actions to ensure the robot’s compliance with safety regulations, paving the way for safer and more widespread adoption of robotic systems in various service domains.

Signed Distance Fields are a common surface representation method widely used for both 3D mapping and obstacle avoidance. While the former traditionally uses projective Truncated Signed Distance Fields (TSDF), the latter often requires a complete Euclidean Signed Distance Field (ESDF) representation of the environment. In this paper, we propose a unified system by combining both methods to generate force vectors to nearby obstacles from a TSDF-based 3D reconstruction. We introduce a new merging scheme to better capture the geometry of the object, with no post-processing requirements, and a way to increase the effective range of the system. Validation experiments demonstrate the accuracy of the force vector calculation by comparing it against an ideal simulated environment. The flexibility of the system is demonstrated by implementing a haptic feedback teleoperation setup, which is validated through a user study in a teleoperation task. Through this, it is shown that the proposed method provides a statistically significant improvement to the task. Finally, a brief description on future improvements to the system is presented.

Non-prehensile transportation of unstable objects presents a challenging task in robotics. To ensure the success of the transportation, it is necessary to consider both the object’s stability via contact dynamics and the motion constraints of the robot. We propose two novel trajectory planning methods derived from sampling and dynamic programming algorithms, tested on a 7-DoF Franka Emika robot against common strategies like Model Predictive Control (MPC) and S-curve planning, particularly under the constraint of a non-rotating tray. The results demonstrate the effectiveness of our methodologies in improving transportation speed. This research contributes to advancements in robotic manipulation techniques by tackling non-prehensile manipulation of dynamically unstable objects.

In this work, an upgraded exoskeleton design is presented with enhanced trajectory tracking and mechanical transparency. Compared to the first version, the design features a 3-DoF actuated shoulder joint and a mechanism to regulate the pretension of Bowden cables. Force/torque sensors are installed to directly measure the interaction forces between the human arm and the exoskeleton at the connecting points. Three control strategies were evaluated to follow a desired trajectory; A PD controller, a PD controller with friction observer, and an adaptive controller based on Radial Basis Function (RBF). These strategies also form the basis for an admittance control, aimed at improving the exoskeleton’s mechanical transparency during interaction with the human arm. Simulations and experimental results demonstrate that the PD control, supported by friction estimation via a momentum observer, achieves superior tracking performance. Moreover, the system’s mechanical transparency is enhanced using the admittance RBF-based controller, showing marginally superior results.

Performing human-like motion is crucial for service humanoid robots. Real-time motion retargeting allows clear observation of the robot’s pose and provides instant feedback during human demonstrator actions. This paper presents an optimization-based real-time anthropomorphic motion retargeting framework for transferring human arm motion to a robot arm. The framework is generic, applicable to both spherical-rotational-spherical (SRS) and non-SRS robot arms. We introduce the normalized normal vector of the arm plane as an anthropomorphic criterion within our framework. The method is validated on a service humanoid robot, with both static and dynamic evaluations. The statistical analysis show that our method maintains strong anthropomorphic features while ensuring accurate wrist pose tracking.

As robots become more and more intelligent, the complexity of the algorithms behind them is increasing. Since these algorithms require high computation power from the onboard robot controller, the weight of the robot and energy consumption increases. A promising solution to tackle this issue is to relocate the expensive computation to the cloud. In this pioneering work, the possibility of relocating a state-of-the-art nonlinear control is investigated. To this end, the Unified Force-Impedance Controller (UFIC) is relocated to a remote location and high frequency feedback loop is established by including the remote controller in the loop. Passivity analysis is used to ensure the stability of the whole system, comprising the robot in interaction with the environment, the communication channel, as well as the remote controller. The instability associated with the communication channel is resolved by Time Domain Passivity Approach (TDPA). The performance of the proposed framework is experimentally evaluated on a robot arm in interaction with the environment. The results illustrate the stability of the system to a time-varying delay of up to 50 ± 10ms.

Tactile sensors are becoming more prevalent in numerous research domains, including robotics, human-robot interaction, and grasping. As the development of customized soft tactile skin for various applications continues to gain momentum, there is an increasing demand for the automation of design and manufacturing processes based on user specifications. Our work presents a partially automated framework for designing and customizing silicone-based skin-like sensors for objects of arbitrary shapes. We assess the performance of stretch and contact sensors featuring custom patterns on complex surfaces, subjecting them to position control, grasping, and manipulation scenarios. Our study's findings demonstrate the feasibility of fabricating skin-like sensors effectively within a semi-automated framework, with potential applications in the aforementioned research domains.

With the increasing demand for new healthcare solutions and technologies, such as those resulting from the COVID-19 crisis, and the growing elderly population, exoskeletons for teleoperation are a promising solution for many future medical applications. In this context, we propose two force- sensitive upper-limb exoskeletons for teleoperation, that are characterized by: i) torque-controlled robotic actuators, ii) rigid-body model compensations, and iii) a lightweight design achieved through the use of Bowden cable transmissions and remotely placed actuators. Specifically, we present a semi-active upper-limb exoskeleton for which we demonstrate human- device interaction control and bilateral teleoperation with force- feedback, evaluated via simulation, in the lab and over the Internet. We also introduce a design for a future fully-active upper-limb exoskeleton with two contact force/torque sensors, for a dual-arm device, which features a novel 3-degrees-of- freedom exoskeleton shoulder design and a contact wrench mitigation controller, as demonstrated through simulation. With this work, we propose the essential technical steps towards a novel teleoperation system for elderly care.

Identifying the inertial parameters of real robotic manipulators is a fundamental step towards realistic modeling and better controller performances, which is crucial for safe human-robot interaction. Our work introduces a novel framework for identifying a generalized set of base inertial parameters of a serial link manipulator. This framework is designed to be adaptable to accommodate any new mounting configuration of the robot. Our theoretical analysis highlights the influence of the robot's mounting configuration on the emergence of new parameters that cannot be identified through the conventional vertical base-axis mounting approach studied previously. To validate our proposed framework, we carried out two main experiments: the first involved simulation to establish the feasibility of our concept, and in the second, our framework was employed on a Franka Emika Robot in a real-world scenario to demonstrate and validate our approach. Our simulation results confirmed the feasibility of our proposed framework, while our real-world experiment successfully identified the generalized base inertial parameter set and validated its applicability to a new robot mounting configuration.

Shared-autonomy control strategies in teleoperation combine human decision-making and robot precision to solve complex tasks. In other words, advanced autonomous control algorithms can compensate for imprecise human commands, reduce the mental workload of the user, and enable the execution of tasks that otherwise wouldn't be feasible. This paper addresses one of these previously challenging scenarios. Herein, we present a novel control framework and motion generator that allows for real-time non-prehensile slosh-free teleoperation of liquids. The proposed approach is able to generate robust trajectories on the follower side which ensures task-space, joint-space, and manipulability constraint satisfaction. Our findings were evaluated through user studies and real-world scenarios. Participants were even explicitly challenged to try to spill liquid through teleoperation, reaching speeds up to 0.6 m/s.

Sophisticated manipulation requires both compliance and accuracy. While tactile robots excel at being compliant, their accuracy is often inadequate for complex manipulation. Contact-rich assembly tasks, such as the insertion and manipulation of objects with small tolerances pose an enormous challenge. Complex, highly integrated assemblies, especially in high-tech areas such as robotics, sensors, or machines, still require human personnel, as they cannot be automated in a satisfactory way. To automate such tasks, especially in the context of labor shortage and Industry 4.0, these limitations must be overcome. Robots need to guarantee force limits for active environments in order to avoid harm or damage. Therefore, in this work, we adapt standard Cartesian impedance control by introducing an integration term for position accuracy and wrench limits for safe compliant interaction with unknown and active environments. We combine this with a virtual energy tank to guarantee the general passivity of the controller. Our controller is benchmarked against standard impedance control for absolute positioning accuracy across the robot workspace. Furthermore, we show its applicability to an industrial insertion task. We demonstrate absolute positioning accuracy (residual error| Ax| < 4e – 4) comparable to rigid robots while preserving compliant behavior.

As labor shortage increases in the health sector, the demand for assistive robotics grows. However, the needed test data to develop those robots is scarce, especially for the application of active 3D object detection, where no real data exists at all. This short paper counters this by introducing such an annotated dataset of real environments. The captured environments represent areas which are already in use in the field of robotic health care research. We further provide ground truth data within one room, for assessing SLAM algorithms running directly on a health care robot.

This paper assesses the usefulness of immersive technology to evaluate perceived discomfort and proxemic behavior towards a robot depending on its size. Therefore, we compared a real and an augmented reality (AR) robot presentation. In a within-subject design, a service humanoid approached participants (N = 32) in four trials in a counterbalanced order. One trial presented a real robot, another showed a same-sized AR version, and two trials showed down-scaled AR versions of the robot. The perceived discomfort and the comfort distance were measured. For the presentation mode comparison, the distance estimation error was measured additionally. The results show that the comfort distance was greater for the AR robot than for the real one. The comfort distance was also greater for the largest robot compared to the smaller sizes, while there was no difference when comparing the smaller ones. There was no difference in perceived discomfort between the presentation modes or the robot sizes. The distance estimation error was greater in AR. The study indicates that results obtained with the AR and real robot are comparable relative to each other. Therefore, utilizing AR could effectively evaluate various versions of robots in terms of the discomfort they induce—a critical prerequisite prior to the manufacturing process. Finally, AR might provide better validity for the evaluation of subjective measures.

Bimanual humanoid assistive robots can be a valuable tool to improve access to physical therapy and multi-dimensional physical status monitoring for older adults living at home. However, at present, there is no implementation of end-effector robot-assisted rehabilitation in assistive social robots. Therefore, this paper illustrates the first steps of a user-centered design approach to develop such a robot for upper limb treatments and rehabilitation. Based on observation of geriatric rehabilitation and expert interviews with physical therapists, an online survey was conducted with 87 physical therapists. The first part of the questionnaire aimed to better understand the context of use, current practices, and goals of geriatric rehabilitation. Our findings suggest that integrating exercises that combine physical and cognitive skills and aim to improve specific activities of daily living (ADLs) are critical. Secondly, a KANO analysis was conducted to prioritize 20 potential features for the robot. Among these features, assist-as-needed (AAN) control for physical exercises, voice control for general settings, and the capability to perform treatments while the patient is seated or lying down were identified as essential or “must-have” features. Third, the possibility of an autonomously conducted weekly assessment of patients’ active range of motion (ROM) and weekly to monthly muscle function testing could allow the best possible monitoring of their functional status. Generally, applying a user-centered design approach allows an interdisciplinary team tasked with developing assistive robots to establish a common objective, based on which an initial prototype can be designed in the next step.

The need for comprehensive telemedicine solutions is becoming increasingly relevant due to challenges associated with the ageing population, the increasing shortage of health-care providers, and, more recently, the global pandemic. Existing solutions primarily focus on, e.g., electronic medical records, audiovisual connections, and, in some cases, robotic systems with very basic capabilities. Here we present a fundamentally new, holistic approach to a remote doctor visit, which enables transparent remote examination, anomaly detection, diagnosis, and rehabilitation. Our dual doctor-patient twin paradigm involves two robotic systems: one representing the doctor to the patient ("GARMI") and one representing the patient to the doctor ("MUCKI"). Through bidirectional telepresence control, this system enables transparent, natural, remote haptic interaction between doctor and patient. The control, interaction, and knowledge transfer to the doctor is enhanced by AI-based visual motion and facial expression analysis as well as a digital twin of the patient. Thus, each stage of a doctor visit can be replicated in the context of telemedicine and shared autonomy: from first assessment to observation-based and remote physical examination, to a better-informed doctor diagnosis and robot-assisted telerehabilitation.

The tactile sense provides local motion information, such as the speed, the direction, and the duration of motion, of an object sliding on the fingertip. An optimal observer should be able to represent the displacement of the object, or the displacement of its own finger on a surface, by integrating these local motion cues over time. In navigation, this skill is known as path integration. We determined human performance for path integration in the fingertip. For this purpose, we develop an experimental task equivalent to the triangle completion task, which is commonly used in locomotion research to study path integration.

In this paper, we investigated depth perception in real and virtual environments when binocular disparity is the sole distance cue. The observers were asked to estimate the relative depth of spheres verbally in virtual and actual environments. Constant apparent sized stimuli were used to measure the just-noticeable difference in depth perception, thus avoiding providing a size gradient cue. Results of the experiments revealed individual differences in virtual reality in contrast to reality. Specifically a subgroup of observers had difficulty perceiving the depth of virtual spheres in virtual reality, which may indicate that they rely on apparent size for judging depth. Furthermore, the just-noticeable differences were more variable in the virtual environment than with real objects. Our results reveal individual differences when the disparity-driven convergence cue is the only distance cue provided in virtual reality.

Using virtual reality for understanding sports performance allows for systematic investigation of human sensorimotor capabilities and meanwhile promotes the design and comparison of realistic immersive platforms. In this paper, we propose a virtual reality-based experimental design for studying the human ability to intercept spinning balls deflected by the Magnus effect. Compared to the previous approaches, we focused on a tight perception-action coupling. Experienced and novice subjects immersed in a 3D soccer stadium were asked to head realistically simulated balls, free kicked with and without sidespin. Consistent with the former studies, qualitative results show that the interception performance systematically relates to both the ball sidespin direction and arrival position for all the subjects, either experienced or not. However, contrary to those former studies where subjects answered only pseudo-verbally, experienced and novice groups differentiate in quantitative performances, supporting that expertise likely appears when perception is coupled to action. Further analyses will be needed to extract the different information-movement relationships governing the behaviors of experienced subjects and novices.