The imitation learning paradigm is a systematic approach for encoding intelligent behaviors into robotic systems. While a model representation of the ideal task behavior can be learned by processing a set of human demonstrations, learning a modeling representation that can generalize the desired behavior to perform well in dynamic environments with human collaborators is an open challenge. To address this problem, we encode intelligent robot behavior as a combination of a popular learned baseline control policy (Gaussian Mixture Model, GMM) with reactive control policies that activate based on triggers from online sensory information during task execution. Two contributions encapsulate the approach: an iterative algorithm to combine the learned and reactive policies and examples for mapping sensory information into desired robot reactive behaviors. The proposed approach was implemented on a bi-manual surgical robot and evaluated on how well the combined control policy balanced the behavioral constraints imposed during a collision avoidance and compliance tasks. Successful dynamic collision avoidance results and compliance responses that reduce environmental forces on the manipulator support the use of this paradigm for designing intelligent robot behaviors which can complement learned models to program complex robot behaviors that can balance task performance in scenarios with human collaborators.

Communication between a human and an intelligent robotic system is an essential component for facilitating effective collaboration. In this paper, we describe a preliminary methodology for creating a collaborative surgical assistant. The proposed framework utilizes Gaussian Mixture Modeling (GMM) of surgical skills. Our approach leverages the GMM to allow for a human to synchronously execute manipulation tasks with the robotic system. Key components of our approach are modified GMM regression techniques which provide the synchronized control policy for the surgical assistant (automatic) manipulator and communicates trajectory information in the form of augmented reality visual cues and haptic guidance proprioceptive feedback to the human operator during task execution. A semi-autonomous surgical knot tie experiment, where the human operator collaborates with an automatic manipulator, was conducted to validate the proof of concept. We show that the variability by the human user allows the team to complete the task and provide insights into future system improvements.

Often in the field of haptic guidance, an important question is how the robotic device should assist some imperfect human movement. While many control strategies have been suggested to help improve human performance in particular tasks, structuring guidance in a generalizable way remains elusive. Many assistive controllers rely on predicting a user's goal movement or knowing some idealized trajectory a-priori but may fail to assist during an arbitrary task. In this study, we propose a ‘flipped’ approach to studying human-robot collaborative behavior - we ask humans to assist a robotic device whose movements are in some way imperfect. We conducted an experiment during which subjects assisted a haptic device performing a shape-following task autonomously but with different types of error-prone controllers. For each shape, we evaluated a simple trajectory-following controller as well as one with human-like motion constraints. We also evaluated the role of visual feedback on a user's ability to help the robot accomplish the unknown task. We found that the human was generally able to improve the robotic error in all trajectories when the robotic motion did not include gravity compensation; however, error reduction was primarily in the vertical direction. When the robotic controller included gravity compensation, the human user was not able to improve errors significantly, except for vertical errors when provided visual feedback. In the no visual feedback conditions, the human user contributed to significantly greater error for most paths compared to a robot with gravity compensation, indicating an inability to provide assistance in that case.

We propose a novel Augmented Reality (AR) Head Mounted Display (HMD) haptic-enabled device which is capable of providing visual and vibrotactile directional cues to locate objects of interest. Using the vibrotactile cues, the device communicates prioritization information to users without the need for additional graphics. This work builds upon a human-robot teaming AR application, AugRE, which provides both situational awareness and control interfaces for any number of ROS-enabled robotic systems. The vibrotactile haptic component developed attaches to the AR-HMD and uses a sequence of vibrations to direct the user to specific objects in their proximity. The visual haptic component does the same by overlaying a holographic arrow on the HMD. We present results from a pilot study and discuss system limitations and research areas that may help direct future development for human-robot teaming applications. Results indicate that visual haptic cues provide the best response times. However, high-frequency vibrotactile haptic cues may be a viable alternative for some tasks where the visual space is already saturated.

In teleoperated robots, such as surgical robots, there is a desire to infer the intent of the operator and provide assistance as needed. This lofty goal is especially challenging when it comes to long-horizon inference. In this paper, we propose leveraging a Transformer-based model to predict the long-horizon trajectory of the master-side manipulators of the da Vinci surgical robot, while also investigating the role of trajectory-based haptic guidance cues as potentially assistive cues. Using the JIGSAW dataset, our model achieved an RMSE Cartesian error of 26.14mm when using the provided gesture labels and 32.13mm without gesture labels for master-side manipulators 1-second-ahead trajectory prediction. We then created resistive and assistive haptic guidance cues with a virtual spring between the current manipulator position and prior or future predicted positions, respectively. Each condition consisted of two levels, defined by 0.5s and Is time horizons. We conducted a preliminary human subject study with 10 subjects to investigate the role of these guidance forces on completion time for a running suturing task. While there are no statistically significant time differences based on type of haptic cue and time-horizon, we observed that the long-horizon resistive guidance had weak significance to improve the mean task performance in a washout trial that immediately followed the haptic condition. We also observed a large decrease in user difficulty ratings for this trial. These results indicate that haptic guidance cues could be leveraged in surgical training, potentially resulting in lasting after-effects on performance once the guidance has been removed.

Gripping and manipulating non-rigid and porous objects is an important challenge for manufacturing. Now there are many problems in handling textile materials from a stack or oriented in space. Therefore, the paper presents the design of an improved Bernoulli gripping device with an anti-vibration insert. The inventive gripper structure allows gripping and manipulating textile materials and partially eliminates the shortcomings present in the classic design of the gripper. A technique for theoretical modeling of a gripping device for textile materials has been developed. This made it possible to determine the rational parameters of the gripping device in terms of maximum attraction. Experimental study of power characteristics of gripping device for textile materials has been carried out. The choice of the thickness of the anti-vibration insert made by the 3D printing method is justified. The advantages of the design include enabling gripping of textile materials of manipulation at different position, orientation and from a longer distance. Influence of supply pressure on the beginning of object vibration is analyzed. Parameters of anti-vibration insert are defined for the operation of gripper without object vibration. Note to Practitioners—This paper was motivated by the problem of gripping and holding textile materials during manufacturing. There are many approaches to gripping a piece of textile material that have high energy consumption or can damage it. This paper proposes the design of the gripper that uses compressed air to lift the textile material at different orientations. Using the proposed theoretical model, one can calculate the lifting force of the gripper at different porosity of the material. Knowing the mass of the material, one can determine which parameters to choose for the gripper based on experimental results. Positive results are highlighted in the capture of porous objects without the loss of force at different orientations. Negative aspects are shown in the formation of vibration of the material when reaching a certain pressure level in the gripper. Future research plans to improve the design of the gripper by modeling the gripper using the finite element method and proposing effective methods of manipulating textile materials.

Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during RAS. We modify the original Transformer architecture to be able to generate the current gesture sequence, future gesture sequence, and future trajectory sequence estimations using only the current kinematic data of the surgical robot end-effectors. We evaluate our proposed models on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) and use Leave-One-User-Out (LOUO) cross validation to ensure generalizability of our results. Our models achieve up to 89.3% gesture recognition accuracy, 84.6% gesture prediction accuracy (1 second ahead) and 2.71mm trajectory prediction error (1 second ahead). Our models are comparable to and able to outperform state-of-the-art methods while using only the kinematic data channel. This approach can enabling near-real time surgical activity recognition and prediction.

An important problem in designing human-robot systems is the integration of human intent and performance in the robotic control loop, especially during complex tasks. Bimanual coordination is a complex human behavior that is critical in many fine motor tasks, including robot-assisted surgery. To fully leverage the capabilities of the robot as an intelligent and assistive agent, online recognition of bimanual coordination could be important. Robotic assistance for a suturing task, for example, will be fundamentally different during phases when the suture is wrapped around the instrument (i.e., making a c-loop), than when the ends of the suture are pulled apart. In this study, we develop an online recognition method of bimanual coordination modes (i.e., the directions and symmetries of right and left hand movements) using geometric descriptors of hand motion. We (1) develop this framework based on ideal trajectories obtained during virtual 2D bimanual path following tasks performed by human subjects operating Geomagic Touch haptic devices, (2) test the offline recognition accuracy of bimanual direction and symmetry from human subject movement trials, and (3) evalaute how the framework can be used to characterize 3D trajectories of the da Vinci Surgical System’s surgeon-side manipulators during bimanual surgical training tasks. In the human subject trials, our geometric bimanual movement classification accuracy was 92.3% for movement direction (i.e., hands moving together, parallel, or away) and 86.0% for symmetry (e.g., mirror or point symmetry). We also show that this approach can be used for online classification of different bimanual coordination modes during needle transfer, making a C loop, and suture pulling gestures on the da Vinci system, with results matching the expected modes. Finally, we discuss how these online estimates are sensitive to task environment factors and surgeon expertise, and thus inspire future work that could leverage adaptive control strategies to enhance user skill during robot-assisted surgery.

Bimanual coordination is critical in many robotic and haptic systems, such as surgical robots and rehabilitation robots. While these systems often incorporate two robotic manipulators for each limb, there may be a missed opportunity to leverage overarching models of human bimanual coordination to improve the way in which the robotic manipulators are controlled and respond to the dynamic human operator. In this paper, we study the influences of several bimanual motion factors (e.g., symmetry and direction) on kinematic human joint-space features and performance outcome task-space features in a user study with eleven subjects and two haptic devices. Additionally, we evaluated the ability to use joint-space features to classify types of bimanual movement, showing the potential for a robotic system to predict how users coordinate their limbs. Three classifiers: (1) likelihood ratio, (2) k-nearest neighbor, and (3) support vector machine, were evaluated for classification accuracy in regards to the factor of number of targets. Likelihood ratio resulted in an accuracy of 79.6% with the majority of correct predictions occurring immediately at the start of movement. The task-space performance results reveal that despite the relative direction of both hands, reaching two targets results in lower performance than a single target, and symmetry alone does not contribute to performance disparity. Also, dimensionless integrated absolute jerk (DIAJ) is an indicator of superior performance for this particular task. Furthermore, these results align with current bimanual coordination theory by showing manual performance disparities are a consequence of task constraints and conceptualization.

The effectiveness of control algorithms for teleoperated systems is typically evaluated through experimental performance measures, post-experimental user surveys, and theoretical analysis. However, none of these methods provide an objective assessment of teleoperation algorithms with respect to the real-time changes of human users during teleoperated tasks in terms of physiological, kinematic, or cognitive metrics. In this study, we recruited subjects to control robotically steered needles in a randomized experiment, using four different teleoperation mappings (joint space control, steering control, and Cartesian space control with and without force feedback). We investigated how the choice of these algorithms affect both performance and user response. Our novel steering control mapping, which mimics hub-centered steering, is significantly correlated with decreased cognitive stress and improved teleoperation performance when compared to joint space control. Overall, user experience and teleoperation performance were significantly improved with Cartesian space control, resulting in faster needle insertion, higher targeting accuracy, lower cognitive load, and smoother movements. Furthermore, while additional haptic feedback in Cartesian space provided an improved performance, it may increase user cognitive workload and muscle fatigue. These results highlight the importance of considering human-centric metrics when designing novel teleoperation strategies for complex systems.

Asymmetric-tip, robotically controlled steerable needles have the potential to improve clinical outcomes for many needle-based procedures by allowing the needle to curve and change direction within biological tissue. Algorithms have previously been developed to change the curvature of the needle trajectory via duty-cycled spinning. However, these algorithms require continuous rotation of the steerable needle, preventing the use of instrumentation such as force-torque sensors and electromagnetic trackers, due cable wind-up issues. In this paper, we present two novel control methods for duty-cycling a steerable needle without the need for continuous rotation: bidirectional duty-cycled spinning and duty-cycled flipping. These algorithms can be implemented on existing robotic needle steering systems without hardware changes. We evaluate our algorithms using a custom hollow steerable needle, with an embedded EM tracker and a force-torque sensor. We compare the path tracking error, needle insertion forces, and needle axial rotation torques of our algorithms and found no significant differences between the two algorithms in terms of tracking error. Duty-cycled flipping has significantly lower mean insertion forces and torques than bidirectional duty-cycled spinning, and differences in insertion force and rotation torque variability were also found. These results may have interesting implications for tissue health.

Robotically steered needles can improve clinical procedures by curving significantly within the body to attain targets and avoid obstacles. Needles that steer by tip asymmetry are nonholonomic systems, which are difficult to control manually (i.e. in joint space) due to under-actuation and unintuitive kinematic constraints. We propose a new teleoperation approach for nonholonomic systems (steerable needles in particular) that allows a user to command the desired position of a robot in Cartesian space and provides force feedback to represent kinematic constraints and the position error of the robot. We performed a user study with a virtual environment to evaluate the effectiveness of Cartesian space teleoperation in comparison to traditional joint space teleoperation, as well as the role of force feedback in Cartesian space teleoperation. Time-to-target and needle insertion length were significantly smaller for Cartesian space control than for joint space control, and when combined with force feedback, Cartesian space control resulted in significantly less targeting error than joint space control. Force feedback during Cartesian space control also reduced tracking error between the user and needle during insertion. Users rated Cartesian space control as easier overall; however, a few subjects felt they had less direct control of the needle.

Insertion velocity, tip asymmetry, and shaft diameter may influence steerable needle insertion paths in soft tissue. In this paper we examine the effects of these variables on needle paths in ex vivo goat liver, and demonstrate practical applications of robotic needle steering for ablation, biopsy, and brachytherapy. All experiments were performed using a new portable needle steering robot that steers asymmetric-tip needles under fluoroscopic imaging. For bevel-tip needles, we found that larger diameter needles resulted in less curvature, i.e. less steerability, confirming previous experiments in artificial tissue. The needles steered with radii of curvature ranging from 3:4 cm (for the most steerable pre-bent needle) to 2:97m (for the least steerable bevel needle). Pre-bend angle significantly affected needle curvature, but bevel angle did not. We hypothesize that biological tissue characteristics such as inhomogeneity and viscoelasticity significantly increase path variability. These results underscore the need for closed-loop image guidance for needle steering in biological tissues with complex internal structure.