Teleoperation of robots in space is challenging due to high latency and limited workspace visibility. Previously, the Interactive Planning and Supervised Execution (IPSE) and Augmented Virtuality systems were developed to reduce failure risk. These tools were visualized on a 3D da Vinci surgical console and operated using the da Vinci manipulators or visualized on conventional monitors and operated with a keyboard and mouse. Experimental studies indicated operator preference for the latter. In this work, we develop a 3D virtual reality (VR) interface for IPSE, implemented on a Meta Quest 2 head-mounted display (HMD), and evaluate it against the prior 2D, keyboard-and-mouse-based interface. The results demonstrate improved operator load with the 3D VR interface, with no decrease in task performance, while also providing cost and portability benefits compared to the conventional 2D interface.

Malaria is a worldwide scourge, and the broad deployment of an effective vaccine would improve the lives of millions of people. A vaccine based on Plasmodium falciparum (PfSPZ) sporozoites extracted from the salivary glands of infected mosquitoes shows significant promise. However, the large-scale industrial production of PfSPZ-based vaccines will benefit from automation of the key step of extracting sporozoites from mosquito salivary glands that is currently performed by manual microdissection. In this work, we demonstrate a robotic system prototype for extracting salivary glands from mosquitoes to streamline vaccine production and reduce the need for operators. In the proposed system, mosquitoes are decapitated in an automated robotic pick-place-decapitate process, then a squeezer apparatus extracts mosquito salivary glands from the body. Mosquito detection and body part localization are performed by computer vision methods. The software allows system operation in simulation and on the robotic hardware, which facilitates subsystem development and integration. Experiments show encouraging results with success rates of 93% in robotic mosquito manipulation and 87.1% in salivary gland extraction. The system has the potential to improve the efficiency of PfSPZ vaccine production with significant gains in throughput and reduction in training times for a highly deskilled initial manual step. Further, this system is expected to pave the way for a more mature future system.

We propose a system for visually monitoring and servoing the cutting of a multi-layer insulation (MLI) blanket that covers the envelope of satellites and spacecraft. The main contributions of this paper are: 1) to propose a model for relating visual features describing the engagement depth of the blade to the force exerted on the MLI blanket by the cutting tool, 2) a blade design and algorithm to reliably detect the engagement depth of the blade inside the MLI, and 3) a servoing mechanism to achieve the desired applied force by monitoring the engagement depth. We present results that validate these contributions by comparing forces estimated from visual feedback to measured forces at the blade. We also demonstrate the robustness of the blade design and vision processing under challenging conditions.

Ground-based teleoperation of robot manipulators for on-orbit servicing of spacecraft represents an example of high-payoff, high-risk operations that are challenging to perform due to high latency communications, with telemetry time delays of several seconds. In these scenarios, confidence of operating without failure is paramount. We report the development of an Interactive Planning and Supervised Execution (IPSE) system that takes advantage of accurate 3D reconstruction of the remote environment to enable operators to plan motions in the virtual world, evaluate and adjust the plan, and then supervise execution with the ability to pause and return to the planning environment at any time. We report the results of an experimental evaluation of a representative on-orbit telerobotic servicing task from NASA's upcoming OSAM-1 mission to refuel a satellite in low earth orbit; specifically, to change the robot tool to acquire the fuel supply line and then to insert it into the satellite fill/drain valve. Results of a pilot study show that the operators preferred, and were more successful with, the IPSE system when compared to a conventional teleoperation implementation.

On-orbit servicing of satellites is complicated by the fact that almost all existing satellites were not designed to be serviced. This creates a number of challenges, one of which is to cut and partially remove the protective thermal blanketing that encases a satellite prior to performing the servicing operation. A human operator on Earth can perform this task telerobotically, but must overcome difficulties presented by the multi-second round-trip telemetry delay between the satellite and the operator and the limited, or even obstructed, views from the available cameras.This paper reports the results of ground-based experiments with trained NASA robot teleoperators to compare our recently-reported augmented virtuality visualization to the conventional camera-based visualization. We also compare the master console of a da Vinci surgical robot to the conventional teleoperation interface. The results show that, for the cutting task, the augmented virtuality visualization can improve operator performance compared to the conventional visualization, but that operators are more proficient with the conventional control interface than with the da Vinci master console.

This paper presents an approach for the detection of moving point targets on high frame-rate image sequences in situations when the spatial signal of the target is swamped by noise. A high frame-rate based moving point target detection framework is proposed, in which a target detector is used for analyzing the time domain evolution of image sequences for distinguishing between background and target, and a statistic based target detection model is used for target detection. The method is evaluated using both simulated and real-world high frame-rate data and we provide a comparison to other widely used point target detection approaches. Our experimental results demonstrate that the proposed framework can be used for robust moving point target detection in very low SNR.

This letter presents an approach for the detection of moving point targets on high-frame-rate image sequences with low spatial resolution and low SNR based on higher order statistical theory. We propose a novel method for analyzing the time-domain evolution of image data for distinguishing between the background and the target in situations when the spatial signal of the target is swamped by noise. Our method is formulated to detect a time-domain transient signal of unknown scale and arrival time in noisy background. We proposed a bispectrum-based model to characterize the temporal behavior of pixels, and the detection ability under different frame rates and SNRs is analyzed. The method is evaluated using both simulated and real-world data, and we provide a comparison to other widely used point target detection approaches. Our experimental results demonstrate that our algorithm can efficiently detect extremely low SNR targets that are virtually invisible to humans based on time-domain analysis of image sequences.

Ground-based teleoperation of robots in space is subject to time delays of several seconds or more. This leads to the use of model-based approaches, where the operator interacts with a model (simulation) of the remote environment and the remote robot attempts to reproduce the results of that interaction. But, it is also desirable for the operator to view (delayed) images from the remote scene. These images, however, are often from one or more monocular cameras mounted on the robot end-effector, which leads to several other problems: unintuitive teleoperation due to the eye-in-hand configuration, limited field of view, and lack of stereo visualization. We present an augmented virtuality interface for teleoperation, which can solve these problems by projecting the real camera images onto a registered 3D model of the environment and allowing the operator to select any desired viewpoint. This approach is suitable when there is at least a partial model of the environment, as is the case for satellite servicing. The proposed method begins with a video survey to register the 3D model to the physical environment, followed by a user interface that presents a stereo visualization of the model, augmented by projections of real camera images onto the model. We quantitatively and qualitatively compare the augmented virtuality images to real camera images taken from the same viewpoint and perform experiments to evaluate the efficacy of the augmented virtuality paradigm for teleoperation. The results suggest that this approach can improve operator situation awareness, potentially leading to better performance, especially when the camera views are unintuitive or limited.

Direct haptic feedback and graphical force feedback have both been hypothesized to improve the performance of robot-assisted surgery. In this study we evaluate the benefits of haptic and graphical force feedback on surgeon performance and tissue exploration behavior during a teleoperated palpation task of artificial tissues. Seven surgeon subjects (four experienced in robot-assisted surgery) used a 7-degree-of-freedom teleoperated surgical robot to identify a comparatively rigid rigid target object (representing a calcified artery) in phantom heart models using the following feedback conditions: (1) direct haptic and graphical feedback, (2) direct haptic only, (3) graphical feedback only, and (4) no feedback. To avoid the problems of force sensing in a minimally invasive surgical environment, we use a position-exchange controller with dynamics compensation for direct haptic feedback and a force estimator displayed via tool-tip tracking bar graph for graphical force feedback. Although the transparency of the system is limited with this approach, results show that direct haptic force feedback minimizes applied forces to the tissue, while coupled haptic and graphical force feedback minimizes subject task error. For experienced surgeons, haptic force feedback substantially reduced task error independent of graphical feedback.

Manual palpation of tissue and organs during a surgical procedure provides clinicians with valuable information for diagnosis and surgical planning. In present-day robotassisted minimally invasive surgery systems, lack of perceptible haptic feedback makes it challenging to detect a tumor in an organ or a calcified artery in heart tissue. This study presents an automated tissue property estimation method and a real-time graphical overlay that allow an operator to discriminate hard and soft tissues. We first evaluate experimentally the properties of an artificial tissue and compare seven possible mathematical tissue models. Self-validation as well as cross-validation confirm that the Hunt-Crossley model best describes the experimentally observed phantom tissue properties and is suitable for our purpose. Second, we present the development of a system in which the phantom tissue is palpated using a teleoperated surgical robot, and the stiffness of the Hunt-Crossly model is estimated in real time by recursive least squares. A real-time visual overlay representing tissue stiffness is created using a hue-saturation-luminance representation on a semi-transparent disc at the tissue surface. Hue depicts the stiffness at a palpated point and saturation is calculated based on distance from the point. A simple interpolation technique creates a continuous stiffness color map. In an experiment, the graphical overlay successfully shows the location of an artificial calcified artery hidden in phantom tissue.

Certain surgical procedures require a high degree of precise manual control within a very restricted area. Retinal surgeries are part of this group of procedures. During vitreoretinal surgery, the surgeon must visualize, using a microscope, an area spanning a few hundreds of microns in diameter and manually correct the potential pathology using direct contact, free hand techniques. In addition, the surgeon must find an effective compromise between magnification, depth perception, field of view, and clarity of view. Pre-operative images are used to locate interventional targets, and also to assess and plan the surgical procedure. This paper proposes a method of fusing information contained in pre-operative imagery, such as fundus and OCT images, with intra-operative video to increase accuracy in finding the target areas. We describe methods for maintaining, in real-time, registration with anatomical features and target areas using image processing. This registration allows us to produce information enhanced displays that ensure that the retinal surgeon is always in visual contact with his/her area of interest.

In this paper some applications of motion analysis are investigated for a compact panoramic optical system (panoramic annular lens). Panoramic image acquisition makes multiple or mechanically controlled camera systems needless for many applications. Panoramic annular lens' main advantage to other omnidirectional monitoring systems is that it is a cheap, small, compact device with no external hyperboloidal, spherical, conical or paraboloidal reflecting surface as in other panoramic optical devices. By converting the annular image captured with an NTSC camera to a rectangular one, we get a low-resolution (2.8 pixels/degrees horizontally and 3 pixels/degree vertically) image. We developed algorithms which can analyze this low-resolution image to yield motion information for surveillance and smoke detection.