Tool tip visualization is an essential component of multiple robotic surgical and interventional procedures. In this paper, we introduce a real-time photoacoustic visual servoing system that processes information directly from raw acoustic sensor data, without requiring image formation or segmentation in order to make robot path planning decisions to track and maintain visualization of tool tips. The performance of this novel deep learning-based visual servoing system is compared to that of a visual servoing system which relies on image formation followed by segmentation to make and execute robot path planning decisions. Experiments were conducted with a plastisol phantom, ex vivo tissue, and a needle as the interventional tool. Needle tip tracking performance with the deep learning-based approach outperformed that of the image-based segmentation approach by 67.7% and 55.3% in phantom and ex vivo tissue, respectively. In addition, the deep learning-based system operated within the frame-rate-limiting 10 Hz laser pulse repetition frequency rate, with mean execution times of 75.2 ms and 73.9 ms per acquisition frame with phantom and ex vivo tissue, respectively. These results highlight the benefits of our new approach to integrate deep learning with robotic systems for improved automation and visual servoing of tool tips.

Ultrasound imaging has been developed for image-guided radiotherapy for tumor tracking, and the flexible array transducer is a promising tool for this task. It can reduce the user dependence and anatomical changes caused by the traditional ultrasound transducer. However, due to its flexible geometry, the conventional delay-and-sum (DAS) beamformer may apply incorrect time delay to the radio-frequency (RF) data and produce B-mode images with considerable defocusing and distortion. To address this problem, we propose a novel end-to-end deep learning approach that may alternate the conventional DAS beamformer when the transducer geometry is unknown. Different deep neural networks (DNNs) were designed to learn the proper time delays for each channel, and they were expected to reconstruct the undistorted high-quality B-mode images directly from RF channel data. We compared the DNN results to the standard DAS beamformed results using simulation and flexible array transducer scan data. With the proposed DNN approach, the averaged full-width-at-half-maximum (FWHM) of point scatters is 1.80 mm and 1.31 mm lower in simulation and scan results, respectively; the contrast-to-noise ratio (CNR) of the anechoic cyst in simulation and phantom scan is improved by 0.79 dB and 1.69 dB, respectively; and the aspect ratios of all the cysts are closer to 1. The evaluation results show that the proposed approach can effectively reduce the distortion and improve the lateral resolution and contrast of the reconstructed B-mode images.

Hysterectomy (i.e., surgical removal of the uterus) requires severing the main blood supply to the uterus (i.e., the uterine arteries) while preserving the nearby, often overlapping, ureters. In this paper, we investigate dual-wavelength and audiovisual photoacoustic imaging-based approaches to visualize and differentiate the ureter from the uterine artery and to provide the real-time information needed to avoid accidental ureteral injuries during hysterectomies. Dual-wavelength 690/750 nm photoacoustic imaging was implemented during laparoscopic and open hysterectomies performed on human cadavers, with a custom display approach designed to visualize the ureter and uterine artery. The proximity of the surgical tool to the ureter was calculated and conveyed by tracking the surgical tool in photoacoustic images and mapping distance to auditory signals. The dual-wavelength display showed up to 10 dB contrast differences between the ureter and uterine artery at three separation distances (i.e., 4 mm, 5 mm, and 6 mm) during the open hysterectomy. During the laparoscopic hysterectomy, the ureter and uterine artery were visualized in the dual-wavelength image with up to 24 dB contrast differences. Distances between the ureter and the surgical tool ranged from 2.47 to 7.31 mm. These results are promising for the introduction of dual-wavelength photoacoustic imaging to differentiate the ureter from the uterine artery, estimate the position of the ureter relative to a surgical tool tip, map photoacoustic-based distance measurements to auditory signals, and ultimately guide hysterectomy procedures to reduce the risk of accidental ureteral injuries.

Cardiac interventional procedures are often performed under fluoroscopic guidance, exposing both the patient and operators to ionizing radiation. To reduce this risk of radiation exposure, we are exploring the use of photoacoustic imaging paired with robotic visual servoing for cardiac catheter visualization and surgical guidance. A cardiac catheterization procedure was performed on two in vivo swine after inserting an optical fiber into the cardiac catheter to produce photoacoustic signals from the tip of the fiber-catheter pair. A combination of photoacoustic imaging and robotic visual servoing was employed to visualize and maintain constant sight of the catheter tip in order to guide the catheter through the femoral or jugular vein, toward the heart. Fluoroscopy provided initial ground truth estimates for 1D validation of the catheter tip positions, and these estimates were refined using a 3D electromagnetic-based cardiac mapping system as the ground truth. The 1D and 3D root mean square errors ranged 0.25-2.28 mm and 1.24-1.54 mm, respectively. The catheter tip was additionally visualized at three locations within the heart: (1) inside the right atrium, (2) in contact with the right ventricular outflow tract, and (3) inside the right ventricle. Lasered regions of cardiac tissue were resected for histopathological analysis, which revealed no laser-related tissue damage, despite the use of 2.98 mJ per pulse at the fiber tip (379.2 mJ/cm2fluence). In addition, there was a 19 dB difference in photoacoustic signal contrast when visualizing the catheter tip pre- and post-endocardial tissue contact, which is promising for contact confirmation during cardiac interventional procedures (e.g., cardiac radiofrequency ablation). These results are additionally promising for the use of photoacoustic imaging to guide cardiac interventions by providing depth information and enhanced visualization of catheter tip locations within blood vessels and within the beating heart.

Deep learning methods are capable of performing sophisticated tasks when applied to a myriad of artificial intelligent (AI) research fields. In this paper, we introduce a novel approach to replace the inherently flawed beamforming step during ultrasound image formation by applying deep learning directly to RF channel data. Specifically, we pose the ultrasound beamforming process as a segmentation problem and apply a fully convolutional neural network architecture to segment anechoic cysts from surrounding tissue. We train our network on a dataset created using the Field II ultrasound simulation software to simulate plane wave imaging with a single insonification angle. We demonstrate the success of our architecture in extracting tissue information directly from the raw channel data, which completely bypasses the beamforming step that would otherwise require multiple insonification angles for plane wave imaging. Our simulated results produce mean Dice coefficient of 0.98 ± 0.02, when measuring the overlap between ground truth cyst locations and cyst locations determined by the network. The proposed approach is promising for developing dedicated deep-learning networks to improve the real-time ultrasound image formation process.

Interventional applications of photoacoustic imaging typically require visualization of point-like targets, such as the small, circular, cross-sectional tips of needles, catheters, or brachytherapy seeds. When these point-like targets are imaged in the presence of highly echogenic structures, the resulting photoacoustic wave creates a reflection artifact that may appear as a true signal. We propose to use deep learning techniques to identify these types of noise artifacts for removal in experimental photoacoustic data. To achieve this goal, a convolutional neural network (CNN) was first trained to locate and classify sources and artifacts in pre-beamformed data simulated with -Wave. Simulations initially contained one source and one artifact with various medium sound speeds and 2-D target locations. Based on 3,468 test images, we achieved a 100% success rate in classifying both sources and artifacts. After adding noise to assess potential performance in more realistic imaging environments, we achieved at least 98% success rates for channel signal-to-noise ratios (SNRs) of -9dB or greater, with a severe decrease in performance below -21dB channel SNR. We then explored training with multiple sources and two types of acoustic receivers and achieved similar success with detecting point sources. Networks trained with simulated data were then transferred to experimental waterbath and phantom data with 100% and 96.67% source classification accuracy, respectively (particularly when networks were tested at depths that were included during training). The corresponding mean ± one standard deviation of the point source location error was 0.40 ± 0.22 mm and 0.38 ± 0.25 mm for waterbath and phantom experimental data, respectively, which provides some indication of the resolution limits of our new CNN-based imaging system. We finally show that the CNN-based information can be displayed in a novel artifact-free image format, enabling us to effectively remove reflection artifacts from photoacoustic images, which is not possible with traditional geometry-based beamforming.

One of the risks of the endonasal approach to skull base surgery is inadvertent damage to one of the two carotid arteries that are located behind the bone being drilled. Photoacoustic imaging, which combines a pulsed laser with an ultrasound receiver probe, has been shown to be able to image blood vessels behind bone. We therefore integrated a photoacoustic imaging system with a telerobotic system, where the pulsed laser is delivered via an optical fiber attached to the drill held by one robot arm and the ultrasound receiver is positioned, at some distance from the drilling site, by another robot arm. This paper describes a new method for accurately determining the safe region for drilling, which is defined by the center-line between the two carotid arteries, and presents the first phantom experiments with this system. The results show that the system can determine the center point with an accuracy better than 2 mm, which suggests that it may be sufficient for clinical scenarios where the two carotid arteries can be within 8 mm of each other.

Photoacoustic (PA) images utilize pulsed lasers and ultrasound transducers to visualize targets with higher optical absorption than the surrounding medium. However, they are susceptible to acoustic clutter and background noise artifacts that obfuscate biomedical structures of interest. We investigated three spatial-angular compounding methods to improve PA image quality for biomedical applications, implemented by combining multiple images acquired as an ultrasound probe was rotated about the elevational axis with the laser beam and target fixed. Compounding with conventional averaging was based on the pose information of each PA image, while compounding with weighted and selective averaging utilized both the pose and image content information. Weighted-average compounding enhanced PA images with the least distortion of signal size, particularly when there were large (i.e., 2.5 mm and 7 (°)) perturbations from the initial probe position. Selective-average compounding offered the best improvement in image quality with up 181, 1665, and 1568 times higher contrast, CNR, and SNR, respectively, compared to the mean values of individual PA images. The three presented spatial compounding methods have promising potential to enhance image quality in multiple photoacoustic applications.

We are investigating the use of photoacoustic (PA) imaging to detect critical structures, such as the carotid artery, that may be located behind the bone being drilled during robot-assisted endonasal transsphenoidal surgery. In this system, the laser is mounted on the drill (via an optical fiber) and the 2D ultrasound (US) probe is placed elsewhere on the skull. Both the drill and the US probe are tracked relative to the patient reference frame. PA imaging provides two advantages compared to conventional B-mode US: (1) the laser penetrates thin layers of bone, and (2) the PA image displays targets that are in the laser path. Thus, the laser can be used to (non-invasively) extend the drill axis, thereby enabling reliable detection of critical structures that may reside in the drill path. This setup creates a challenging alignment problem, however, because the US probe must be placed so that its image plane intersects the laser line in the neighborhood of the target anatomy (as estimated from preoperative images). This paper reports on a navigation system developed to assist with this task, and the results of phantom experiments that demonstrate that a critical structure can be detected with an accuracy of approximately 1 mm relative to the drill tip.

Image-guided radiation therapy (IGRT) involves two main procedures, performed in different rooms on different days: (1) treatment planning in the simulator room on the first day, and (2) radiotherapy in the linear accelerator room over multiple subsequent days. Both the simulator and the linear accelerator include CT imaging capabilities, which enables both treatment planning and reproducible patient setup, but does not provide good soft tissue contrast or allow monitoring of the target during treatment. We propose a cooperatively-controlled robot to reproducibly position an ultrasound (US) probe on the patient during simulation and treatment, thereby improving soft tissue visualization and allowing real-time monitoring of the target. A key goal of the robotic system is to produce consistent tissue deformations for both CT and US imaging, which simplifies registration of these two modalities. This paper presents the robotic system design and describes a novel control algorithm that employs virtual springs to implement guidance virtual fixtures during "hands on" cooperative control.