With the advancement of industrial manufacturing and an increase in introduction of robots in the workspace, the need of safe operation, communication and information sharing is paramount. The work presented here focuses on cyber-physical system integration through Digital Twin (DT) technology. Our novel DT architecture is based on a model-free Neuro-Adaptive controller (NAC), and an edge-computing scheme for scene monitoring. The NAC can account for varying robot dynamics in both real and virtual environments, and allows for the DT system to expand the realm of cyber-physical integration without expensive model tuning. The edge-computing device introduced in our architecture, observes the robot's workspace from a distance with a wider field of view. This wide viewpoint, enhances the detection and mitigation of any obstacles entering the robot's workspace during operation. We experimentally evaluated the performance of our proposed architecture by introducing dynamic obstacles during a pick-and-place task that both the physical robot and its digital twin had to avoid. Results show that the proposed DT architecture successfully integrates the novel controller and edge-computing elements and successfully performs the given navigation task. The results also show that NAC outperforms a PD controller with more than 70% improvement in joint tracking error between the physical and virtual robots. It was observed that the latency experienced while using NAC is about 48 % lower than when Proportional-Derivative (PD) controller was operational.

Digital Twin technology is being envisioned to be an integral part of the industrial evolution in modern generation. With the rapid advancement in the Internet-of-Things (IoT) technology and increasing trend of automation, integration between the virtual and the physical world is now realizable to produce practical digital twins. However, the existing definitions of digital twin is incomplete and sometimes ambiguous. Herein, we conduct historical review and analyze the modern generic view of digital twin to create its new extended definition. We also review and discuss the existing work in digital twin in safety-critical robotics applications. Especially, the usage of digital twin in industrial applications necessitates autonomous and remote operations due to environmental challenges. However, the uncertainties in the environment may need close monitoring and quick adaptation of the robots which need to be safety-proof and cost effective. We demonstrate a case study on developing a framework for safety-critical robotic arm applications and present the system performance to show its advantages, and discuss the challenges and scopes ahead.

This paper presents an attention-based, deep learning framework that converts robot camera frames with dynamic content into static frames to more easily apply simultaneous localization and mapping (SLAM) algorithms. The vast majority of SLAM methods have difficulty in the presence of dynamic objects appearing in the environment and occluding the area being captured by the camera. Despite past attempts to deal with dynamic objects, challenges remain to reconstruct large, occluded areas with complex backgrounds. Our proposed Dynamic-GAN framework employs a generative adversarial network to remove dynamic objects from a scene and inpaint a static image free of dynamic objects. The Dynamic-GAN framework utilizes spatial-temporal transformers, and a novel spatial-temporal loss function. The evaluation of Dynamic-GAN was comprehensively conducted both quantitatively and qualitatively by testing it on benchmark datasets, and on a mobile robot in indoor navigation environments. As people appeared dynamically in close proximity to the robot, results showed that large, feature-rich occluded areas can be accurately reconstructed with our attention-based deep learning framework for dynamic object removal. Through experiments we demonstrate that our proposed algorithm has up to 25% better performance on average as compared to the standard benchmark algorithms.

In this paper, we propose a novel online algorithm for motion similarity measurements during human-robot interaction (HRI). Specifically, we formulate a Segment-based Online Dynamic Time Warping (SODTW) algorithm that can be used for understanding of repeated and cyclic human motions, in the context of rehabilitation or social interaction. The algorithm can estimate both the human-robot motion similarity and the time delay to initiate motion and combine these values as a metric to adaptively select appropriate robot imitation repertoires. We validated the algorithm offline by post-processing experimental data collected from a cohort of 55 subjects during imitation episodes with our social robot Zeno. Furthermore, we implemented the algorithm online on Zeno and collected further experimental results with 13 human subjects. These results show that the algorithm can reveal important features of human movement including the quality of motion and human reaction time to robot stimuli. Moreover, the robot can adapt to appropriate human motion speeds based on similarity measurements calculated using this algorithm, enabling future adaptive rehabilitation interventions for conditions such as Autism Spectrum Disorders (ASD).

The importance of tactile sensing for physical human–robot interaction (pHRI) and dexterous manipulation is well known. SkinSim is a novel simulation environment for tactile robot skins in which one can study design tradeoffs involved in deploying whole-body, dense sensor arrays. In this article, scalable modeling approaches are presented for simulating pressure-sensitive robot skin patches, with simultaneous consideration of sensing element geometry and mechanical structure, signal quality, data processing, and closed-loop force controller performance. The open-source simulation architecture of SkinSim is compatible with Gazebo and robot operating system (ROS) programming environments and supports both robot skin dynamic models, as well as tactile sensing element models. An experimentally validated force dispersion model was introduced for the simulation of sensors embedded in a mechanical damping layer. Simulation examples of robot skin with different tactile resolutions are presented using parameter values extracted from an experimental testbed. Thus, simulation results were experimentally validated and the skin sensor density impact on a simple pHRI controller performance was evaluated. Performance measures include center of pressure (COP) estimation error and control signal settling time, overshoot, and steady-state errors. Results suggest that while COP errors decrease in denser sensor skins, controller performance also deteriorates. Therefore, optimal robot skin designs will have to consider application-dependent tradeoffs. Similar results were confirmed in simulation with a larger skin patch containing approximately 4000 tactels and deployed on the end-effector of a collaborative mobile manipulator.Note to Practitioners—This article was motivated by the expensive and time-consuming process of designing tactile skin for robots. When designing pressure-sensitive whole-body sensor arrays for physical human–robot interaction, there are tradeoffs related to sensor resolution, accuracy, and response time, among others. Instead of building prototypes and evaluating them experimentally, the SkinSim simulation environment allows automatic testing of skins with simultaneous consideration of sensing element geometry and mechanical structure, signal quality, data processing, and closed-loop force controller performance. The user can specify several configurations to test and thereby explore the best tradeoffs before actually prototyping any hardware.

With the rise of collaborative robots, the need for safe, reliable, and efficient physical human-robot interaction (pHRI) has grown. High-performance pHRI requires robust and stable controllers suitable for multiple degrees of freedom (DoF) and highly nonlinear robots. In this article, we describe a cascade-loop pHRI controller, which relies on human force and pose measurements and can adapt to varying robot dynamics online. It can also adapt to different users and simplifies the interaction by making the robot behave according to a prescribed dynamic model. In our controller formulation, two neural networks (NNs) in the “outer-loop” predict human motion intent and estimate a reference trajectory for the robot that the “inner-loop” controller follows. The inner-loop imposes a prescribed error dynamics (PED) with the help of a model-free neuroadaptive controller (NAC), which uses a NN to feedback linearize the robot dynamics. Lyapunov stability analysis gives weight tuning laws that guarantee that the error signals are bounded and the desired reference trajectory is achieved. Our control scheme was implemented on a Personal Robot 2 robot and validated through an exploratory experimental study in point-to-point collaborative motion. Results indicate fast convergence of our controller, and the resulting tracking error, motion jerk, and human control effort are comparable with other methods that require prior training, knowledge, and calibration.

In this paper, we introduce a novel application of social robotics in healthcare: high fidelity, facially expressive, robotic patient simulators (RPSs), and explore their usage within a clinical experimental context. Current commercially-available RPSs, the most commonly used humanoid robots worldwide, are substantially limited in their usability and fidelity due to the fact that they lack one of the most important clinical interaction and diagnostic tools: an expressive face. Using autonomous facial synthesis techniques, we synthesized pain both on a humanoid robot and comparable virtual avatar. We conducted an experiment with 51 clinicians and 51 laypersons (n = 102), to explore differences in pain perception across the two groups, and also to explore the effects of embodiment (robot or avatar) on pain perception. Our results suggest that clinicians have lower overall accuracy in detecting synthesized pain in comparison to lay participants. We also found that all participants are overall less accurate detecting pain from a humanoid robot in comparison to a comparable virtual avatar, lending support to other recent findings in the HRI community. This research ultimately reveals new insights into the use of RPSs as a training tool for calibrating clinicians' pain detection skills.