Michael C. Yip

VLDB 2026 Research / reviewers in the wild / expert

dblp:78/7424 · also Mike C. Yip · DBLP profile ↗

← Back

66ranked-venue papers

6as first author

43since 2021 · last 2025

0000-0001-9689-0172ORCID · corroborated

Domains — the database's venue-derived domains; a paper can count in several

Artificial intelligence and machine learning · 48 · 2 first-author · 34 since 2021Systems, architecture and hardware · 42 · 2 first-author · 31 since 2021Applied, interdisciplinary, general and emerging computing · 15 · 4 first-author · 7 since 2021Graphics, computer vision, multimedia, augmented reality and games · 5 · 1 first-author · 3 since 2021Human-computer interaction and ubiquitous computing · 2 · 1 since 2021

Year	Publication	Venue	Position
2025	MEDiC: Autonomous Surgical Robotic Assistance to Maximizing Exposure for Dissection and Cautery abstract Surgical automation has the capability to improve the consistency of patient outcomes and broaden access to advanced surgical care in underprivileged communities. Shared autonomy, where the robot automates routine subtasks while the surgeon retains partial teleoperative control, offers great potential to make an impact. In this paper we focus on one important skill within surgical shared autonomy: Automating robotic assistance to maximize visual exposure and apply tissue tension for dissection and cautery. Ensuring consistent exposure to visualize the surgical site is crucial for both efficiency and patient safety. However, achieving this is highly challenging due to the complexities of manipulating deformable volumetric tissues that are prevalent in surgery. To address these challenges we propose MEDiC, a framework for autonomous surgical robotic assistance to Maximizing Exposure for Dissection and Cautery. We integrate a differentiable physics model with perceptual feedback to achieve our two key objectives: 1) Maximizing tissue exposure and applying tension for a specified dissection site through visual-servoing control and 2) Selecting optimal control positions for a dissection target based on deformable Jacobian analysis. We quantitatively assess our method through repeated real robot experiments on a tissue phantom. Our visual-servoing and optimal control position selection achieve success rate of 100% and 82% respectively in ablation study. We also showcase our framework's capabilities through dissection experiments using shared autonomy on real animal tissue. Chung-Pang Wang, Nikhil Shinde, Fei Liu 0033, Florian Richter 0002, Michael C. Yip	ICRA	6
2025	AutoPeel: Adhesion-Aware Safe Peeling Trajectory Optimization for Robotic Wound Care abstract Chronic wounds, including diabetic ulcers, pressure ulcers, and ulcers secondary to venous hypertension, affects more than 6.5 million patients and a yearly cost of more than $25 billion in the United States alone. Chronic wound treatment is currently a manual process, and we envision a future where robotics and automation will aid in this treatment to reduce cost and improve patient care. In this work, we present the development of the first robotic system for wound dressing removal which is reported to be the worst aspect of living with chronic wounds. Our method leverages differentiable physics-based simulation to perform gradient-based trajectory optimization for peeling trajectory planning. By integrating fracture mechanics of adhesion, we are able to model the peeling effect inherent to dressing adhesion. The system is further guided by carefully designed objective functions that promote both efficient and safe control, reducing the risk of tissue damage. We validated the efficacy of our approach through a series of experiments conducted on both synthetic skin phantoms and real human subjects. Our results demonstrate the system's ability to achieve precise and safe dressing removal trajectories, offering a promising solution for automating this essential healthcare procedure. Youcheng Zhang, Fei Liu 0033, Florian Richter 0002, Michael C. Yip	ICRA	5
2025	CtRNet-X: Camera-to-Robot Pose Estimation in Real-World Conditions using a Single Camera abstract Camera-to-robot calibration is crucial for visionbased robot control and requires effort to make it accurate. Recent advancements in markerless pose estimation methods have eliminated the need for time-consuming physical setups for camera-to-robot calibration. While the existing markerless pose estimation methods have demonstrated impressive accuracy without the need for cumbersome setups, they rely on the assumption that all the robot joints are visible within the camera's field of view. However, in practice, robots usually move in and out of view, and some portion of the robot may stay out-of-frame during the whole manipulation task due to real-world constraints, leading to a lack of sufficient visual features and subsequent failure of these approaches. To address this challenge and enhance the applicability to visionbased robot control, we propose a novel framework capable of estimating the robot pose with partially visible robot manipulators. Our approach leverages the Vision-Language Models for fine-grained robot components detection, and integrates it into a keypoint-based pose estimation network, which enables more robust performance in varied operational conditions. The framework is evaluated on both public robot datasets and self-collected partial-view datasets to demonstrate our robustness and generalizability. As a result, this method is effective for robot pose estimation in a wider range of realworld manipulation scenarios. Jingpei Lu, Zekai Liang, Tristin Xie, Florian Richter 0002, Sainan Liu, Michael C. Yip	ICRA	7
2025	Haptic Shoulder for Rendering Biomechanically Accurate Joint Limits for Human-Robot Physical Interactions abstract Human-robot physical interaction (pHRI) is a rapidly evolving research field with significant implications for physical therapy, search and rescue, and telemedicine. However, a major challenge lies in accurately understanding human constraints and safety in human-robot physical experiments without an IRB and physical human experiments. Concerns regarding human studies include safety concerns, repeatabil-ity, scalability of the number, and diversity of participants. This paper examines whether a physical approximation can serve as a stand-in for human subjects to enhance robot autonomy for physical assistance. This paper introduces the SHULDRD (Shoulder Haptic Universal Limb Dynamic Repositioning Device), an economical and anatomically similar device designed for real-time testing and deploying pHRI planning tasks on robots in the real world. SHULDRD replicates human shoulder motion, providing crucial force feedback and safety data. The device's open-source CAD and software facilitate easy construction and use, ensuring broad accessibility for researchers. By providing a flexible platform able to emulate infinite human subjects, ensure repeatable trials, and provide quantitative metrics to assess the effectiveness of the robotic intervention, SHULDRD aims to improve the safety and efficacy of human-robot physical interactions. Project URL: https://sites.google.com/view/haptic-shoulder/home Elizabeth Peiros, Calvin Joyce, Tarun Murugesan, Roger Nguyen, Isabella Fiorini, Rizzi Galibut, Michael C. Yip	ICRA	7
2025	KineDepth: Utilizing Robot Kinematics for Online Metric Depth Estimation abstract Depth perception is essential for a robot’s spatial and geometric understanding of its environment, with many tasks traditionally relying on hardware-based depth sensors like RGB-D or stereo cameras. However, these sensors face practical limitations, including issues with transparent and reflective objects, high costs, calibration complexity, spatial and energy constraints, and increased failure rates in compound systems. While monocular depth estimation methods offer a cost-effective and simpler alternative, their adoption in robotics is limited due to their output of relative rather than metric depth, which is crucial for robotics applications. In this paper, we propose a method that utilizes a single calibrated camera, enabling the robot to act as a "measuring stick" to convert relative depth estimates into metric depth in real-time as tasks are performed. Our approach employs an LSTM-based metric depth regressor, trained online and refined through probabilistic filtering, to accurately restore the metric depth across the monocular depth map, particularly in areas proximal to the robot’s motion. Experiments with real robots demonstrate that our method significantly outperforms current state-of-the-art monocular metric depth estimation techniques, achieving a 22.1% reduction in depth error and a 52% increase in success rate for a downstream task. Soofiyan Atar, Yuheng Zhi, Florian Richter 0002, Michael C. Yip	IROS	4
2025	Differentiable Rendering-based Pose Estimation for Surgical Robotic Instruments abstract Robot pose estimation is a challenging and crucial task for vision-based surgical robotic automation. Typical robotic calibration approaches, however, are not applicable to surgical robots, such as the da Vinci Research Kit (dVRK) [1], due to joint angle measurement errors from cable-drives and the partially visible kinematic chain. Hence, previous works in surgical robotic automation used tracking algorithms to estimate the pose of the surgical tool in real-time and compensate for the joint angle errors. However, a big limitation of these previous tracking works is the initialization step which relied on only keypoints and SolvePnP. In this work, we fully explore the potential of geometric primitives beyond just keypoints with differentiable rendering, cylinders, and construct a versatile pose matching pipeline in a novel pose hypothesis space. We demonstrate the state-of-the-art performance of our single-shot calibration method with both calibration consistency and real surgical tasks. As a result, this marker-less calibration approach proves to be a robust and generalizable initialization step for surgical tool tracking. Zekai Liang, Zih-Yun Chiu, Florian Richter 0002, Michael C. Yip	IROS	4
2025	Optimal Motion Scaling for Delayed Telesurgery abstract Robotic teleoperation over long communication distances poses challenges due to delays in commands and feedback from network latency. One simple yet effective strategy to reduce errors and increase performance under delay is to downscale the relative motion between the operating surgeon and the robot. The question remains as to what is the optimal scaling factor, and how this value changes depending on the level of latency as well as operator tendencies. We present user studies investigating the relationship between latency, scaling factor, and performance. The results of our studies demonstrate a statistically significant difference in performance between users and across scaling factors for certain levels of delay. These findings indicate that the optimal scaling factor for a given level of delay is specific to each user, motivating the need for personalized models for optimal performance. We present techniques to model the user-specific mapping of latency level to scaling factor for optimal performance, leading to an efficient and effective solution to optimizing performance of robotic teleoperation and specifically telesurgery under large communication delay. Jason Lim, Florian Richter 0002, Zih-Yun Chiu, Jaeyon Lee, Ethan Quist, Nathan Fisher, Jonathan Chambers, Steven Hong, Michael C. Yip	IROS	9
2025	BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction Using Neural Radiance Fields abstract Reconstruction of deformable scenes from endoscopic videos is important for many applications such as intraop-erative navigation, surgical visual perception, and robotic surgery. It is a foundational requirement for realizing autonomous robotic interventions for minimally invasive surgery. However, previous approaches in this domain have been limited by their modular nature and are confined to specific camera and scene settings. Our work adopts the Neural Radiance Fields (NeRF) approach to learning 3D implicit representations of scenes that are both dynamic and deformable over time, and furthermore with unknown cam-era poses. This work removes the constraints of known cam-era poses and overcomes the drawbacks of the state-of-the-art unstructured dynamic scene reconstruction technique, which relies on the static part of the scene for accurate re-construction. Through several experimental datasets, we demonstrate the versatility of our proposed model to adapt to diverse camera and scene settings, and show its promise for both current and future robotic surgical systems. Shreya Saha, Zekai Liang, Jingpei Lu, Michael C. Yip, Sainan Liu	WACV	5
2024	Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives abstract Controlling robotic manipulators via visual feedback requires a known coordinate frame transformation between the robot and the camera. Uncertainties in mechanical systems as well as camera calibration create errors in this coordinate frame transformation. These errors result in poor localization of robotic manipulators and create a significant challenge for applications that rely on precise interactions between manipulators and the environment. In this work, we estimate the camera-to-base transform and joint angle measurement errors for surgical robotic tools using an image based insertion-shaft detection algorithm and probabilistic models. We apply our proposed approach in both a structured environment as well as an unstructured environment and measure to demonstrate the efficacy of our methods. Christopher D'Ambrosia, Florian Richter 0002, Zih-Yun Chiu, Nikhil Shinde, Fei Liu 0033, Henrik I. Christensen, Michael C. Yip	ICRA	7
2024	Zero-Shot Constrained Motion Planning Transformers Using Learned Sampling Dictionaries abstract Constrained robot motion planning is a ubiquitous need for robots interacting with everyday environments, but it is a notoriously difficult problem to solve. Many sampled points in a sample-based planner need to be rejected as they fall outside the constraint manifold, or require significant iterative effort to correct. Given this, few solutions exist that present a constraint-satisfying trajectory for robots, in reasonable time and of low path cost. In this work, we present a transformer-based model for motion planning with task space constraints for manipulation systems. Vector Quantized-Motion Planning Transformer (VQ-MPT) is a recent learning-based model that reduces the search space for unconstrained planning for sampling-based motion planners. We propose to adapt a pre-trained VQMPT model to reduce the search space for constraint planning without retraining or finetuning the model. We also propose to update the neural network output to move sampling regions closer to the constraint manifold. Our experiments show how VQ-MPT improves planning times and accuracy compared to traditional planners in simulated and real-world environments. Unlike previous learning methods, which require task-related data, our method uses pre-trained neural network models and requires no additional data for training and finetuning the model making this a one-shot process. We also tested our method on a physical Franka Panda robot with real-world sensor data, demonstrating the generalizability of our algorithm. We anticipate this approach to be an accessible and broadly useful for transfering learned neural planners to various robotic-environment interaction scenarios. Jacob J. Johnson, Ahmed H. Qureshi, Michael C. Yip	ICRA	3
2024	Real-to-Sim Deformable Object Manipulation: Optimizing Physics Models with Residual Mappings for Robotic Surgery abstract Accurate deformable object manipulation (DOM) is essential for achieving autonomy in robotic surgery, where soft tissues are being displaced, stretched, and dissected. Many DOM methods can be powered by simulation, which ensures realistic deformation by adhering to the governing physical constraints and allowing for model prediction and control. However, real soft objects in robotic surgery, such as membranes and soft tissues, have complex, anisotropic physical parameters that a simulation with simple initialization from cameras may not fully capture. To use the simulation techniques in real surgical tasks, the real-to-sim gap needs to be properly compensated. In this work, we propose an online, adaptive parameter tuning approach for simulation optimization that (1) bridges the real-to-sim gap between a physics simulation and observations obtained 3D perceptions through estimating a residual mapping and (2) optimizes its stiffness parameters online. Our method ensures a small residual gap between the simulation and observation and improves the simulation’s predictive capabilities. The effectiveness of the proposed mechanism is evaluated in the manipulation of both a thin-shell and volumetric tissue, representative of most tissue scenarios. This work contributes to the advancement of simulation-based deformable tissue manipulation and holds potential for improving surgical autonomy. Fei Liu 0033, Yuelei Li, Michael C. Yip	ICRA	6
2024	Tracking Snake-Like Robots in the Wild Using Only a Single Camera abstract Robot navigation within complex environments requires precise state estimation and localization to ensure robust and safe operations. For ambulating mobile robots like robot snakes, traditional methods for sensing require multiple embedded sensors or markers, leading to increased complexity, cost, and increased points of failure. Alternatively, deploying an external camera in the environment is very easy to do, and marker-less state estimation of the robot from this camera’s images is an ideal solution: both simple and cost-effective. However, the challenge in this process is in tracking the robot under larger environments where the cameras may be moved around without extrinsic calibration, or maybe when in motion (e.g., a drone following the robot). The scenario itself presents a complex challenge: single-image reconstruction of robot poses under noisy observations. In this paper, we address the problem of tracking ambulatory mobile robots from a single camera. The method combines differentiable rendering with the Kalman filter. This synergy allows for simultaneous estimation of the robot’s joint angle and pose while also providing state uncertainty which could be used later on for robust control. We demonstrate the efficacy of our approach on a snake-like robot in both stationary and non-stationary (moving) cameras, validating its performance in both structured and unstructured scenarios. The results achieved show an average error of 0.05 m in localizing the robot’s base position and 6 degrees in joint state estimation. We believe this novel technique opens up possibilities for enhanced robot mobility and navigation in future exploratory and search-and-rescue missions. Jingpei Lu, Florian Richter 0002, Michael C. Yip	ICRA	4
2024	Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration abstract Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io. Abigail O'Neill, Abhiram Maddukuri, Abhishek Gupta 0004, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew E. Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu 0003, Charlotte Le, Chelsea Finn, Chen Wang 0053, Chenfeng Xu, Cheng Chi 0001, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Drieß, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Paul Foster, Fangchen Liu, Federico Ceola, Fei Xia 0002, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su 0001, Haoshu Fang, Henghui Bao, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim 0001, Jaimyn Drake, Jan Peters 0001, Jan Schneider 0007, Jasmine Hsu, Jeannette Bohg, Jeffrey T. Bingham, Jensen Gao, Jiaheng Hu, Jiajun Wu 0001, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan 0001, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Kenneth Y. Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Zhang 0002, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Yunliang Chen 0001, Lerrel Pinto, Li Fei-Fei 0001, Liam Tan, Linxi Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang 0003, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma 0001, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen 0001, Nicolas Heess, Nikhil J. Joshi, Niko Sünderhauf, Norman Di Palo, Nur Muhammad Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R. Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu 0010, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Rafael Rafailov, Ria Doshi, Roberto Martin Martin, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante-Gomez, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham D. Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair 0003, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wolfram Burgard, Xiaolong Wang 0004, Xinghao Zhu, Xinyang Geng, Liangwei Xu, Yecheng Jason Ma 0001, Yejin Kim 0003, Yevgen Chebotar, Yilin Wu 0003, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang 0001, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zichen Jeff Cui, Zichen Zhang 0016, Zipeng Lin	ICRA	159
2024	AnyOKP: One-Shot and Instance-Aware Object Keypoint Extraction with Pretrained ViT abstract Towards flexible object-centric visual perception, we propose a one-shot instance-aware object keypoint (OKP) extraction approach, AnyOKP, which leverages the powerful representation ability of pretrained vision transformer (ViT), and can obtain keypoints on multiple object instances of arbitrary category after learning from a support image. An off-the-shelf petrained ViT is directly deployed for generalizable and transferable feature extraction, which is followed by training-free feature enhancement. The best-prototype pairs (BPPs) are searched for in support and query images based on appearance similarity, to yield instance-unaware candidate keypoints. Then, the entire graph with all candidate keypoints as vertices are divided into sub-graphs according to the feature distributions on the graph edges. Finally, each sub-graph represents an object instance. AnyOKP is evaluated on real object images collected with the cameras of a robot arm, a mobile robot, and a surgical robot, which not only demonstrates the cross-category flexibility and instance awareness, but also show remarkable robustness to domain shift and viewpoint change. Fangbo Qin, Taogang Hou, Michael C. Yip	ICRA	5
2024	Achieving Autonomous Cloth Manipulation with Optimal Control via Differentiable Physics-Aware Regularization and Safety Constraints abstract Cloth manipulation is a category of deformable object manipulation of great interest to the robotics community, from applications of automated laundry-folding and home organizing to textiles and flexible manufacturing. Despite the desire for automated cloth manipulation, the thin-shell dynamics and under-actuation nature of cloth present significant challenges for robots to effectively interact with them. Many recent works omit explicit modeling in favor of learning-based methods that may yield control policies directly. However, these methods require large training sets that must be collected and curated. In this regard, we create a framework for differentiable modeling of cloth dynamics leveraging an Extended Position-based Dynamics (XPBD) algorithm. Together with the desired control objective, physics-aware regularization terms are designed for better results, including trajectory smoothness and elastic potential energy. In addition, safety constraints, such as avoiding obstacles, can be specified using signed distance functions (SDFs). We formulate the cloth manipulation task with safety constraints as a constrained optimization problem, which can be effectively solved by mainstream gradient-based optimizers thanks to the end-to-end differentiability of our framework. Finally, we assess the framework with various safety thresholds and demonstrate the feasibility of result trajectories on a surgical robot. The effects of the regularization terms are analyzed in an additional ablation study. Fei Liu 0033, Michael C. Yip	ICRA	4
2024	SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Data abstract A major source of endoscopic tissue tracking errors during deformations stems from wrong data association between observed sensor measurements with previously tracked scene. To mitigate this issue, we present a surgical perception framework, SuPerPM, that leverages learning-based non-rigid point cloud matching for data association, thus accommodating larger deformations than previous approaches which relied on Iterative Closest Point (ICP) for point associations. The learning models typically require training data with ground truth point cloud correspondences, which is challenging or even impractical to collect in surgical environments. Thus, for tuning the learning model, we gather endoscopic data of soft tissue being manipulated by a surgical robot and then establish correspondences between point clouds at different time points to serve as ground truth. This was achieved by employing a position-based dynamics (PBD) simulation to ensure that the correspondences adhered to physical constraints. The proposed framework is demonstrated on several challenging surgical datasets that are characterized by large deformations, achieving superior performance over advanced surgical scene tracking algorithms.1 Albert J. Miao, Ali Alabiad, Fei Liu 0033, Jingpei Lu, Florian Richter 0002, Michael C. Yip	IROS	8
2024	SURESTEP: An Uncertainty-Aware Trajectory Optimization Framework to Enhance Visual Tool Tracking for Robust Surgical Automation abstract Inaccurate tool localization is one of the main reasons for failures in automating surgical tasks. Imprecise robot kinematics and noisy observations caused by the poor visual acuity of an endoscopic camera make tool tracking challenging. Previous works in surgical automation adopt environment-specific setups or hard-coded strategies instead of explicitly considering motion and observation uncertainty of tool tracking in their policies. In this work, we present SURESTEP, an uncertainty-aware trajectory optimization framework for robust surgical automation.We model the uncertainty in tool tracking by considering noise sources that are typical in surgical environments.Using a Gaussian assumption to propagate our uncertainty models through a given tool trajectory, SURESTEP provides a general framework that minimizes the upper bound on the entropy of the final estimated tool distribution.We showcase our method by performing the first-ever, to our knowledge, needle regrasping with a moving endoscopic camera.We compare SURESTEP with a baseline method on a real-world suture needle regrasping task under challenging environmental conditions, such as poor lighting and a moving endoscopic camera. The results over 60 regrasps on the da Vinci Research Kit (dVRK) demonstrate that our optimized trajectories significantly outperform the un-optimized baseline. Nikhil Shinde, Zih-Yun Chiu, Florian Richter 0002, Jason Lim, Yuheng Zhi, Sylvia L. Herbert, Michael C. Yip	IROS	7
2024	Tracking and mapping in medical computer vision: A review abstract As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation. Adam Schmidt, Omid Mohareri, Simon P. DiMaio, Michael C. Yip, Tim Salcudean	Medical Image Anal.	4
2023	Markerless Camera-to-Robot Pose Estimation via Self-Supervised Sim-to-Real Transfer abstract Solving the camera-to-robot pose is a fundamental requirement for vision-based robot control, and is a process that takes considerable effort and cares to make accurate. Traditional approaches require modification of the robot via markers, and subsequent deep learning approaches enabled markerless feature extraction. Mainstream deep learning methods only use synthetic data and rely on Domain Randomization to fill the sim-to-real gap, because acquiring the 3D annotation is labor-intensive. In this work, we go beyond the limitation of 3D annotations for real-world data. We propose an end-to-end pose estimation framework that is capable of online camera-to-robot calibration and a self-supervised training method to scale the training to unlabeled real-world data. Our framework combines deep learning and geometric vision for solving the robot pose, and the pipeline is fully differentiable. To train the Camera-to-Robot Pose Estimation Network (CtRNet), we leverage foreground segmentation and differentiable rendering for image-level self-supervision. The pose prediction is visualized through a renderer and the image loss with the input image is back-propagated to train the neural network. Our experimental results on two public real datasets confirm the effectiveness of our approach over existing works. We also integrate our framework into a visual servoing system to demonstrate the promise of real-time precise robot pose estimation for automation tasks. Jingpei Lu, Florian Richter 0002, Michael C. Yip	CVPR	3
2023	Real-Time Constrained 6D Object-Pose Tracking of An In-Hand Suture Needle for Minimally Invasive Robotic Surgery abstract Autonomous suturing has been a long-sought-after goal for surgical robotics. Outside of staged environments, accurate localization of suture needles is a critical foundation for automating various suture needle manipulation tasks in the real world. When localizing a needle held by a gripper, previous work usually tracks them separately without considering their relationship. Because of the significant errors that can arise in the stereo-triangulation of objects and instruments, their reconstructions may often not be consistent. This can lead to unrealistic tool-needle grasp reconstructions that are infeasible. Instead, an obvious strategy to improve localization would be to leverage constraints that arise from contact, thereby constraining reconstructions of objects and instruments into a jointly feasible space. In this work, we consider feasible grasping constraints when tracking the 6D pose of an in-hand suture needle. We propose a reparameterization trick to define a new state space for describing a needle pose, where grasp constraints can be easily defined and satisfied. Our proposed state space and feasible grasping constraints are then incorporated into Bayesian filters for real-time needle localization. In the experiments, we show that our constrained methods outperform previous unconstrained tracking approaches and demonstrate the importance of incorporating feasible grasping constraints into automating suture needle manipulation tasks. Zih-Yun Chiu, Florian Richter 0002, Michael C. Yip	ICRA	3
2023	Suture Thread Spline Reconstruction from Endoscopic Images for Robotic Surgery with Reliability-driven Keypoint Detection abstract Automating the process of manipulating and delivering sutures during robotic surgery is a prominent problem at the frontier of surgical robotics, as automating this task can significantly reduce surgeons' fatigue during tele-operated surgery and allow them to spend more time addressing higher-level clinical decision making. Accomplishing autonomous suturing and suture manipulation in the real world requires accurate suture thread localization and reconstruction, the process of creating a 3D shape representation of suture thread from 2D stereo camera surgical image pairs. This is a very challenging problem due to how limited pixel information is available for the threads, as well as their sensitivity to lighting and specular reflection. We present a suture thread reconstruction work that uses reliable keypoints and a Minimum Variation Spline (MVS) smoothing optimization to construct a 3D centerline from a segmented surgical image pair. This method is comparable to previous suture thread reconstruction works, with the possible benefit of increased accuracy of grasping point estimation. Our code and datasets will be available at: https://github.com/ucsdarclab/thread-reconstruction. Neelay Joglekar, Fei Liu 0033, Ryan K. Orosco, Michael C. Yip	ICRA	4
2023	Design and Mechanics of Cable-Driven Rolling Diaphragm Transmission for High-Transparency Robotic Motion abstract Applications of rolling diaphragm transmissions for medical and teleoperated robotics are of great interest, due to the low friction of rolling diaphragms combined with the power density and stiffness of hydraulic transmissions. However, the stiffness-enabling pressure preloads can form a tradeoff against bearing loading in some rolling diaphragm layouts, and transmission setup can be difficult. Utilization of cable drives compliment the rolling diaphragm transmission's advantages, but maintaining cable tension is crucial for optimal and consistent performance. In this paper, a coaxial opposed rolling diaphragm layout with cable drive and an electronic transmission control system are investigated, with a focus on system reliability and scalability. Mechanical features are proposed which enable force balancing, decoupling of transmission pressure from bearing loads, and maintenance of cable tension. Key considerations and procedures for automation of transmission setup, phasing, and operation are also presented. We also present an analysis of system stiffness to identify key compliance contributors, and conduct experiments to validate prototype design performance. Hoi Man Lam, W. Jared Walker, Lucas Jonasch, Dimitri A. Schreiber, Michael C. Yip	ICRA	5
2023	Mobility Analysis of Screw-Based Locomotion and Propulsion in Various Media abstract Robots “in-the-wild” encounter and must traverse widely varying terrain, ranging from solid ground to granular materials like sand to full liquids. Numerous approaches exist, including wheeled and legged robots, each excelling in specific domains. Screw-based locomotion is a promising approach for multi-domain mobility, leveraged in exploratory robotic designs, including amphibious vehicles and snake robotics. However, unlike other forms of locomotion, there is limited exploration of the models, parameter effects, and efficiency for multi-terrain Archimedes screw locomotion. In this work, we present work towards this missing component in understanding screw-based locomotion: comprehensive experimental results and performance analysis across different media. We designed a mobile test bed for indoor and outdoor experimentation to collect this data. Beyond quantitatively showing the multi-domain mobility of screw-based locomotion, we envision future researchers and engineers using the presented results to design effective screw-based locomotion systems. Jason Lim, Calvin Joyce, Elizabeth Peiros, Mingwei Yeoh, Peter V. Gavrilov, Sara G. Wickenhiser, Dimitri A. Schreiber, Florian Richter 0002, Michael C. Yip	ICRA	9
2023	Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking abstract Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensive surgical per-ception framework, Semantic-SuPer, that integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes, benefiting downstream tasks like surgical navigation. The proposed frame-work is demonstrated on challenging endoscopic data with deforming tissue, showing its advantages over our baseline and several other state-of-the-art approaches. Our code and dataset are available at https://github.com/ucsdarclab/Python-SuPer. Albert J. Miao, Jingpei Lu, Shunkai Yu, Zih-Yun Chiu, Florian Richter 0002, Michael C. Yip	ICRA	7
2023	Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering abstract State estimation from measured data is crucial for robotic applications as autonomous systems rely on sensors to capture the motion and localize in the 3D world. Among sensors that are designed for measuring a robot's pose, or for soft robots, their shape, vision sensors are favorable because they are information-rich, easy to set up, and cost-effective. With recent advancements in computer vision, deep learning-based methods no longer require markers for identifying feature points on the robot. However, learning-based methods are data-hungry and hence not suitable for soft and prototyping robots, as building such bench-marking datasets is usually infeasible. In this work, we achieve image-based robot pose estimation and shape reconstruction from camera images. Our method requires no precise robot meshes, but rather utilizes a differentiable renderer and primitive shapes. It hence can be applied to robots for which CAD models might not be available or are crude. Our parameter estimation pipeline is fully differentiable. The robot shape and pose are estimated iteratively by back-propagating the image loss to update the parameters. We demonstrate that our method of using geometrical shape primitives can achieve high accuracy in shape reconstruction for a soft continuum robot and pose estimation for a robot manipulator. Jingpei Lu, Fei Liu 0033, Cédric Girerd, Michael C. Yip	ICRA	4
2023	Finding Biomechanically Safe Trajectories for Robot Manipulation of the Human Body in a Search and Rescue Scenario abstract There has been increasing awareness of the difficulties in reaching and extracting people from mass casualty scenarios, such as those arising from natural disasters. While platforms have been designed to consider reaching casualties and even carrying them out of harm's way, the challenge of repositioning a casualty from its found configuration to one suitable for extraction has not been explicitly explored. Furthermore, this planning problem needs to incorporate biomechanical safety considerations for the casualty. Thus, we present a first solution to biomechanically safe trajectory generation for repositioning limbs of unconscious human casualties. We describe biomechanical safety as mathematical constraints, mechanical descriptions of the dynamics for the robot-human coupled system, and the planning and trajectory optimization process that considers this coupled and constrained system. We finally evaluate our approach over several variations of the problem and demonstrate it on a real robot and human subject. This work provides a crucial part of search and rescue that can be used in conjunction with past and present works involving robots and vision systems designed for search and rescue. Elizabeth Peiros, Zih-Yun Chiu, Yuheng Zhi, Nikhil Shinde, Michael C. Yip	IROS	5
2023	Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning abstract Reinforcement learning (RL) agents have long sought to approach the efficiency of human learning. Humans are great observers who can learn by aggregating external knowledge from various sources, including observations from others' policies of attempting a task. Prior studies in RL have incorporated external knowledge policies to help agents improve sample efficiency. However, it remains non-trivial to perform arbitrary combinations and replacements of those policies, an essential feature for generalization and transferability. In this work, we present Knowledge-Grounded RL (KGRL), an RL paradigm fusing multiple knowledge policies and aiming for human-like efficiency and flexibility. We propose a new actor architecture for KGRL, Knowledge-Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding-based attentive action prediction. KIAN also addresses entropy imbalance, a problem arising in maximum entropy KGRL that hinders an agent from efficiently exploring the environment, through a new design of policy distributions. The experimental results demonstrate that KIAN outperforms alternative methods incorporating external knowledge policies and achieves efficient and flexible learning. Our implementation is available at https://github.com/Pascalson/KGRL.git . Zih-Yun Chiu, Yi-Lin Tuan, William Yang Wang, Michael C. Yip	NeurIPS	4
2022	Image Based Reconstruction of Liquids from 2D Surface Detections abstract In this work, we present a solution to the challenging problem of reconstructing liquids from image data. The challenges in reconstructing liquids, which is not faced in previous reconstruction works on rigid and deforming surfaces, lies in the inability to use depth sensing and color features due the variable index of refraction, opacity, and environmental reflections. Therefore, we limit ourselves to only surface detections (i.e. binary mask) of liquids as observations and do not assume any prior knowledge on the liquids properties. A novel optimization problem is posed which reconstructs the liquid as particles by minimizing the error between a rendered surface from the particles and the surface detections while satisfying liquid constraints. Our solvers to this optimization problem are presented and no training data is required to apply them. We also propose a dynamic prediction to seed the reconstruction optimization from the previous time-step. We test our proposed methods in simulation and on two new liquid datasets which we open source11https://github.com/ucsdarclab/liquid_reconstruction so the broader research community can continue developing in this under explored area. Florian Richter 0002, Ryan K. Orosco, Michael C. Yip	CVPR	3
2022	CRANE: a 10 Degree-of-Freedom, Tele-surgical System for Dexterous Manipulation within Imaging Bores abstract Physicians perform minimally invasive percuta-neous procedures under Computed Tomography (CT) image guidance both for the diagnosis and treatment of numerous diseases. For these procedures performed within Computed Tomography Scanners, robots can enable physicians to more accurately target sub-dermal lesions while increasing safety. However, existing robots for this application have limited dexterity, workspace, or accuracy. This paper describes the design, manufacture, and performance of a highly dexterous, low-profile, 8+2 Degree-of-Freedom (DoF) robotic arm for CT guided percutaneous needle biopsy. In this article, we propose CRANE: CT Robot and Needle Emplacer. The design focuses on system dexterity with high accuracy: extending physicians' ability to manipulate and insert needles within the scanner bore while providing the high accuracy possible with a robot. We also propose and validate a system architecture and control scheme for low profile and highly accurate image-guided robotics, that meets the clinical requirements for target accuracy during an in-situ evaluation. The accuracy is additionally evaluated through a trajectory tracking evaluation resulting in$< \boldsymbol{0.2}\mathbf{mm}$and$<\boldsymbol{ 0.71}^{\circ}$tracking error. Finally, we present a novel needle driving and grasping mechanism with controlling electronics that provides simple manufacturing, sterilization, and adaptability to accom-modate different sizes and types of needles. Dimitri A. Schreiber, Zhaowei Yu, Hanpeng Jiang, Taylor Henderson, Guosong Li, Julie Yu, Renjie Zhu, Alexander M. Norbash, Michael C. Yip	ICRA	9
2022	Markerless Suture Needle 6D Pose Tracking with Robust Uncertainty Estimation for Autonomous Minimally Invasive Robotic Surgery abstract Suture needle localization is necessary for autonomous suturing. Previous approaches in autonomous suturing often relied on fiducial markers rather than markerless detection schemes for localizing a suture needle due to the in-consistency of markerless detections. However, fiducial markers are not practical for real-world applications and can often be occluded from environmental factors in surgery (e.g., blood). Therefore in this work, we present a robust tracking approach for estimating the 6D pose of a suture needle when using inconsistent detections. We define observation models based on suture needles' geometry that captures the uncertainty of the detections and fuse them temporally in a probabilistic fashion. In our experiments, we compare different permutations of the observation models in the suture needle localization task to show their effectiveness. Our proposed method outperforms previous approaches in localizing a suture needle. We also demonstrate the proposed tracking method in an autonomous suture needle regrasping task and ex vivo environments**The code is available at https://github.com/ucsdarclab/suture-needle-tracking.. Zih-Yun Chiu, Albert Z. Liao, Florian Richter 0002, Bjorn Johnson, Michael C. Yip	IROS	5
2022	Autonomous Navigation in Unknown Environments With Sparse Bayesian Kernel-Based Occupancy Mapping abstract This article focuses on online occupancy mapping and real-time collision checking onboard an autonomous robot navigating in a large unknown environment. Commonly used voxel and octree map representations can be easily maintained in a small environment but have increasing memory requirements as the environment grows. We propose a fundamentally different approach for occupancy mapping, in which the boundary between occupied and free space is viewed as the decision boundary of a machine learning classifier. This work generalizes a kernel perceptron model which maintains a very sparse set of support vectors to represent the environment boundaries efficiently. We develop a probabilistic formulation based on relevance vector machines, handling measurement noise, and probabilistic occupancy classification, supporting autonomous navigation. We provide an online training algorithm, updating the sparse Bayesian map incrementally from streaming range data, and an efficient collision-checking method for general curves, representing potential robot trajectories. The effectiveness of our mapping and collision checking algorithms is evaluated in tasks requiring autonomous robot navigation and active mapping in unknown environments. Thai Duong 0001, Michael C. Yip, Nikolay Atanasov 0001	IEEE Trans. Robotics	2
2022	Constrained Motion Planning Networks X abstract Constrained motion planning is a challenging field of research, aiming for computationally efficient methods that can find a collision-free path on the constraint manifolds between a given start and goal configuration. These planning problems come up surprisingly frequently, such as in robot manipulation for performing daily life assistive tasks. However, few solutions to constrained motion planning are available, and those that exist struggle with high computational time complexity in finding a path solution on the manifolds. To address this challenge, we present Constrained Motion Planning Networks X (CoMPNetX). It is a neural planning approach, comprising a conditional deep neural generator and discriminator with neural gradients-based fast projection operator. We also introduce neural task and scene representations conditioned on which the CoMPNetX generates implicit manifold configurations to turbo-charge any underlying classical planner such as sampling-based motion planning methods for quickly solving complex constrained planning tasks. We show that our method finds path solutions with high success rates and lower computation times than state-of-the-art traditional path-finding tools on various challenging scenarios. Ahmed H. Qureshi, Jiangeng Dong, Asfiya Baig, Michael C. Yip	IEEE Trans. Robotics	4
2022	ARCSnake: Reconfigurable Snakelike Robot With Archimedean Screw Propulsion for Multidomain Mobility abstract Exploring and navigating in extreme environments, such as caves, oceans, and planetary bodies, are often too hazardous for humans, and as such, robots are possible surrogates. These robots are met with significant locomotion challenges that require traversing a wide range of surface roughnesses and topologies. Previous locomotion strategies, involving wheels or ambulatory motion, such as snake platforms, have success on specific surfaces but fail in others, which could be detrimental in exploration and navigation missions. In this article, we present a novel approach that combines snakelike robots with an Archimedean screw locomotion mechanism to provide multiple, effective mobility strategies in a large range of environments, including those that are difficult to traverse for wheeled and ambulatory robots. This work develops a robotic system called ARCSnake to demonstrate this locomotion principle and tested it in a variety of different terrains and environments in order to prove its controllable, multidomain, navigation capabilities. These tests show a wide breadth of scenarios that ARCSnake can handle, hence demonstrating its ability to traverse through extreme terrains. Florian Richter 0002, Peter V. Gavrilov, Hoi Man Lam, Amir Degani, Michael C. Yip	IEEE Trans. Robotics	5
2022	Robotic Tool Tracking Under Partially Visible Kinematic Chain: A Unified Approach abstract Anytime a robot manipulator is controlled via visual feedback, the transformation between the robot and camera frame must be known. However, in the case where cameras can only capture a portion of the robot manipulator in order to better perceive the environment being interacted with, there is greater sensitivity to errors in calibration of the base-to-camera transform. A secondary source of uncertainty during robotic control are inaccuracies in joint angle measurements which can be caused by biases in positioning and complex transmission effects such as backlash and cable stretch. In this work, we bring together these two sets of unknown parameters into a unified problem formulation when the kinematic chain is partially visible in the camera view. We prove that these parameters are nonidentifiable implying that explicit estimation of them is infeasible. To overcome this, we derive a smaller set of parameters we call lumped error since it lumps together the errors of calibration and joint angle measurements. A particle filter method is presented and tested in simulation and on two real world robots to estimate the lumped error and show the efficiency of this parameter reduction. Florian Richter 0002, Jingpei Lu, Ryan K. Orosco, Michael C. Yip	IEEE Trans. Robotics	4
2022	DiffCo: Autodifferentiable Proxy Collision Detection With Multiclass Labels for Safety-Aware Trajectory Optimization abstract The objective of trajectory optimization algorithms is to achieve an optimal collision-free path between start and goal states. In real-world scenarios, where environments can be complex and nonhomogeneous, a robot needs to be able to gauge whether a state will be in collision with various objects in order to meet some safety metrics. The collision detector should be computationally efficient and, ideally, analytically differentiable to facilitate stable and rapid gradient descent during optimization. However, methods today lack an elegant approach to detect collision differentiably, relying rather on numerical gradients that can be unstable. We present DiffCo, the first, fully autodifferentiable, nonparametric model for collision detection. Its nonparametric behavior allows one to compute collision boundaries on the fly and update them, requiring no pretraining and allowing it to update continuously in dynamic environments. It provides robust gradients for trajectory optimization via backpropagation and is often 10–100 times faster to compute than its geometric counterparts. DiffCo also extends trivially to modeling different object collision classes for semantically informed trajectory optimization. Yuheng Zhi, Nikhil Das, Michael C. Yip	IEEE Trans. Robotics	3
2021	ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring abstract Traumatic injuries require timely intervention, but medical expertise is not always available at the patient’s location. Despite recent advances in telecommunications, surgeons still have limited tools to remotely help inexperienced surgeons. Mixed Reality hints at a future where remote collaborators work side-by-side as if co-located; however, we still do not know how current technology can improve remote surgical collaboration. Through role-playing and iterative-prototyping, we identify collaboration practices used by expert surgeons to aid novice surgeons as well as technical requirements to facilitate these practices. We then introduce ARTEMIS, an AR-VR collaboration system that supports these key practices. Through an observational study with two expert surgeons and five novice surgeons operating on cadavers, we find that ARTEMIS supports remote surgical mentoring of novices through synchronous point, draw, and look affordances and asynchronous video clips. Most participants found that ARTEMIS facilitates collaboration despite existing technology limitations explored in this paper. Danilo Gasques, Janet G. Johnson, Thomas Sharkey, Yuanyuan Feng, Ru Wang 0002, Zhuoqun Robin Xu, Enrique Zavala, Wanze Xie, Konrad Davis, Michael C. Yip, Nadir Weibel	CHI	12
2021	Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning abstract Regrasping a suture needle is an important yet time-consuming process in suturing. To bring efficiency into regrasping, prior work either designs a task-specific mechanism or guides the gripper toward some specific pick-up point for proper grasping of a needle. Yet, these methods are usually not deployable when the working space is changed. Therefore, in this work, we present rapid trajectory generation for bimanual needle regrasping via reinforcement learning (RL). Demonstrations from a sampling-based motion planning algorithm is incorporated to speed up the learning. In addition, we propose the ego-centric state and action spaces for this bimanual planning problem, where the reference frames are on the end-effectors instead of some fixed frame. Thus, the learned policy can be directly applied to any feasible robot configuration. Our experiments in simulation show that the success rate of a single pass is 97%, and the planning time is 0.0212s on average, which outperforms other widely used motion planning algorithms. For the real-world experiments, the success rate is 73.3% if the needle pose is reconstructed from an RGB image, with a planning time of 0.0846s and a run time of 5.1454s. If the needle pose is known beforehand, the success rate becomes 90.5%, with a planning time of 0.0807s and a run time of 2.8801s. Zih-Yun Chiu, Florian Richter 0002, Emily K. Funk, Ryan K. Orosco, Michael C. Yip	ICRA	5
2021	Data-driven Actuator Selection for Artificial Muscle-Powered Robots abstract Even though artificial muscles have gained popularity due to their compliant, flexible and compact properties, there currently does not exist an easy way of making informed decisions on the appropriate actuation strategy when designing a muscle-powered robot; thus limiting the transition of such technologies into broader applications. What’s more, when a new muscle actuation technology is developed, it is difficult to compare it against existing robot muscles. To accelerate the development of artificial muscle applications, we propose a data-driven approach for robot muscle actuator selection using Support Vector Machines (SVM). This first-of-its-kind method gives users insight into which actuators fit their specific needs and actuation performance criteria, making it possible for researchers and engineers with little to no prior knowledge of artificial muscles to focus on application design. It also provides a platform to benchmark existing, new or yet-to-be discovered artificial muscle technologies. We test our method on unseen existing robot muscle designs to prove its usability on real-world applications. We provide an open-access, web-searchable interface for easy access to our models that will additionally allow for the continuous contribution of new actuator data from groups around the world to enhance and expand these models. Taylor Henderson, Yuheng Zhi, Angela Liu, Michael C. Yip	ICRA	4
2021	Model-Predictive Control of Blood Suction for Surgical Hemostasis using Differentiable Fluid Simulations abstract Recent developments in surgical robotics have led to new advancements in the automation of surgical sub-tasks such as suturing, soft tissue manipulation, tissue tensioning and cutting. However, integration of dynamics to optimize these control policies for the variety of scenes encountered in surgery remains unsolved. Towards this effort, we investigate the integration of differentiable fluid dynamics to optimizing a suction tool’s trajectory to clear the surgical field from blood as fast as possible. The fully differentiable fluid dynamics is integrated with a novel suction model for effective model predictive control of the tool. The differentiability of the fluid model is crucial because we utilize the gradients of the fluid states with respect to the suction tool position to optimize the trajectory. Through a series of experiments, we demonstrate how, by incorporating fluid models, the trajectories generated by our method can perform as good as or better than handcrafted human-intuitive suction policies. We also show that our method is adaptable and can work in different cavity conditions while using a single handcrafted strategy fails. Jingbin Huang, Fei Liu 0033, Florian Richter 0002, Michael C. Yip	ICRA	4
2021	Real-to-Sim Registration of Deformable Soft Tissue with Position-Based Dynamics for Surgical Robot Autonomy abstract Autonomy in robotic surgery is very challenging in unstructured environments, especially when interacting with deformable soft tissues. The main difficulty is to generate model-based control methods that account for deformation dynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes within the scene, however, model-based controllers integrated with dynamic properties, a more accurate and safe approach, has not been studied before. Considering the mechanic coupling between the robot and the environment, it is crucial to develop a registered, simulated dynamical model. In this work, we propose an online, continuous, real-to-sim registration method to bridge 3D visual perception with position-based dynamics (PBD) modeling of tissues. The PBD method is employed to simulate soft tissue dynamics as well as rigid tool interactions for model-based control. Meanwhile, a vision-based strategy is used to generate 3D reconstructed point cloud surfaces based on real-world manipulation, so as to register and update the simulation. To verify this real-to-sim approach, tissue experiments have been conducted on the da Vinci Research Kit. Our real-to-sim approach successfully reduces registration error online, which is especially important for safety during autonomous control. Moreover, it achieves higher accuracy in occluded areas than fusion-based reconstruction. Fei Liu 0033, Yunhai Han, Jingpei Lu, Florian Richter 0002, Michael C. Yip	ICRA	6
2021	SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction abstract Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue tracking and surgical tool tracking processes. By leveraging transfer learning, the deep-learning-based approach requires minimal training data and reduced feature engineering efforts to fully perceive a surgical scene. The framework was tested on three publicly available datasets, which use the da Vinci® Surgical System, for comprehensive analysis. Experimental results show that our framework achieves state-of-the-art tracking performance in a surgical environment by utilizing deep learning for feature extraction. Jingpei Lu, Ambareesh Jayakumari, Florian Richter 0002, Yang Li 0041, Michael C. Yip	ICRA	5
2021	Optimal Multi-Manipulator Arm Placement for Maximal Dexterity during Robotics Surgery abstract Robot arm placements are oftentimes a limitation in surgical preoperative procedures, relying on trained staff to evaluate and decide on the optimal positions for the arms. Given new and different patient anatomies, it can be challenging to make an informed choice, leading to more frequently colliding arms or limited manipulator workspaces. In this paper, we develop a method to generate the optimal manipulator base positions for the multi-port da Vinci surgical system that minimizes self-collision and environment-collision, and maximizes the surgeon’s reachability inside the patient. Scoring functions are defined for each criterion so that they may be optimized over. Since for multi-manipulator setups, a large number of free parameters are available to adjust the base positioning of each arm, a challenge becomes how one can expediently assess possible setups. We thus also propose methods that perform fast queries of each measure with the use of a proxy collision-checker. We then develop an optimization method to determine the optimal position using the scoring functions. We evaluate the optimality of the base positions for the robot arms on canonical trajectories, and show that the solution yielded by the optimization program can satisfy each criterion. The metrics and optimization strategy are generalizable to other surgical robotic platforms so that patient-side manipulator positioning may be optimized and solved. James Di, Nikhil Das, Michael C. Yip	ICRA	4
2021	Motion Planning Networks: Bridging the Gap Between Learning-Based and Classical Motion Planners abstract This article describes motion planning networks (MPNet), a computationally efficient, learning-based neural planner for solving motion planning problems.MPNet uses neural networks to learn general near-optimal heuristics for path planning in seen and unseen environments. It takes environment information such as raw point cloud from depth sensors, as well as a robot's initial and desired goal configurations and recursively calls itself to bidirectionally generate connectable paths. In addition to finding directly connectable and near-optimal paths in a single pass, we show that worst-case theoretical guarantees can be proven if we merge this neural network strategy with classical sample-based planners in a hybrid approach while still retaining significant computational and optimality improvements. To train the MPNet models, we present an active continual learning approach that enables MPNet to learn from streaming data and actively ask for expert demonstrations when needed, drastically reducing data for training. We validate MPNet against gold-standard and state-of-the-art planning methods in a variety of problems from two-dimensional to seven-dimensional robot configuration spaces in challenging and cluttered environments, with results showing significant and consistently stronger performance metrics, and motivating neural planning in general as a modern strategy for solving motion planning problems efficiently. Ahmed H. Qureshi, Yinglong Miao, Anthony Simeonov, Michael C. Yip	IEEE Trans. Robotics	4
2020	Composing Task-Agnostic Policies with Deep Reinforcement Learning Ahmed H. Qureshi, Jacob J. Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, Michael C. Yip	ICLR	6
2020	Autonomous Navigation in Unknown Environments using Sparse Kernel-based Occupancy Mapping abstract This paper focuses on real-time occupancy mapping and collision checking onboard an autonomous robot navigating in an unknown environment. We propose a new map representation, in which occupied and free space are separated by the decision boundary of a kernel perceptron classifier. We develop an online training algorithm that maintains a very sparse set of support vectors to represent obstacle boundaries in configuration space. We also derive conditions that allow complete (without sampling) collision-checking for piecewise-linear and piecewise-polynomial robot trajectories. We demonstrate the effectiveness of our mapping and collision checking algorithms for autonomous navigation of an Ackermann-drive robot in unknown environments. Thai Duong 0001, Nikhil Das, Michael C. Yip, Nikolay Atanasov 0001	ICRA	3
2020	ARCSnake: An Archimedes' Screw-Propelled, Reconfigurable Serpentine Robot for Complex Environments abstract This paper presents the design and performance of a new locomotion strategy for serpentine robots using screw propulsion. The ARCSnake robot comprises serially linked, identical modules, each incorporating an Archimedes' screw for propulsion and a universal joint (U-Joint) for orientation control. When serially chained, these modules form a versatile serpentine robot platform which enables the robot to reshape its body configuration for varying environments, typical of a snake. Furthermore, the Archimedes' screws allow for novel omni-wheel drive-like motions by speed controlling their screw threads. This paper considers the mechanical and electrical design, as well as the software architecture for realizing a fully integrated system. The system includes 3N actuators for N segments, each controlled using a BeagleBone Black with a customized power-electronics cape, a 9 Degrees of Freedom (DoF) Inertial Measurement Unit (IMU), and a scalable communication channel over ROS. This robot serves as the first proof-of-concept demonstration of the NASA-JPL Exobiology Extant Life Surveyor (EELS) program that aims to deliver scientific instrumentation deep within the plume vents, caves, and ice sheets of Enceladus and Europa in search for extant lifeforms*. Dimitri A. Schreiber, Florian Richter 0002, Andrew Bilan, Peter V. Gavrilov, Hoi Man Lam, Casey H. Price, Kalind C. Carpenter, Michael C. Yip	ICRA	8
2020	Dynamically Constrained Motion Planning Networks for Non-Holonomic Robots abstract Reliable real-time planning for robots is essential in today's rapidly expanding automated ecosystem. In such environments, traditional methods that plan by relaxing constraints become unreliable or slow-down for kinematically constrained robots. This paper describes the algorithm Dynamic Motion Planning Networks (Dynamic MPNet), an extension to Motion Planning Networks, for non-holonomic robots that address the challenge of real-time motion planning using a neural planning approach. We propose modifications to the training and planning networks that make it possible for real-time planning while improving the data efficiency of training and trained models' generalizability. We evaluate our model in simulation for planning tasks for a non-holonomic robot. We also demonstrate experimental results for an indoor navigation task using a Dubins car. Jacob J. Johnson, Linjun Li, Fei Liu 0033, Ahmed H. Qureshi, Michael C. Yip	IROS	5
2020	Learning-Based Proxy Collision Detection for Robot Motion Planning Applications abstract This article demonstrates that collision detection-intensive applications such as robotic motion planning may be accelerated by performing collision checks with a machine learning model. We propose Fastron, a learning-based algorithm, to model a robot's configuration space to be used as a proxy collision detector in place of standard geometric collision checkers. We demonstrate that leveraging the proxy collision detector results in up to an order of magnitude faster performance in robot simulation and planning than state-of-the-art collision detection libraries. Our results show that Fastron learns a model more than 100 times faster than a competing C-space modeling approach, while also providing theoretical guarantees of learning convergence. Using the open motion planning libraries (OMPLs), we were able to generate initial motion plans across all experiments with varying robot and environment complexities and workspace obstacle locations. With Fastron, we can repeatedly generate new motion plans at a 56 Hz rate, showing its application toward autonomous surgical assistance task in shared environments with human-controlled manipulators. All performance gains were achieved despite using only CPU-based calculations, suggesting further computational gains with a GPU approach that can parallelize tensor algebra. Code is available online. Nikhil Das, Michael C. Yip	IEEE Trans. Robotics	2
2019	Adversarial Imitation via Variational Inverse Reinforcement Learning Ahmed H. Qureshi, Byron Boots, Michael C. Yip	ICLR (Poster)	3
2019	Motion Planning Networks abstract Fast and efficient motion planning algorithms are crucial for many state-of-the-art robotics applications such as self-driving cars. Existing motion planning methods become ineffective as their computational complexity increases exponentially with the dimensionality of the motion planning problem. To address this issue, we present Motion Planning Networks (MPNet), a neural network-based novel planning algorithm. The proposed method encodes the given workspaces directly from a point cloud measurement and generates the end-to-end collision-free paths for the given start and goal configurations. We evaluate MPNet on various 2D and 3D environments including the planning of a 7 DOF Baxter robot manipulator. The results show that MPNet is not only consistently computationally efficient in all environments but also generalizes to completely unseen environments. The results also show that the computation time of MPNet consistently remains less than 1 second in all presented experiments, which is significantly lower than existing state-of-the-art motion planning algorithms. Ahmed H. Qureshi, Anthony Simeonov, Mayur J. Bency, Michael C. Yip	ICRA	4
2019	Motion Scaling Solutions for Improved Performance in High Delay Surgical Teleoperation abstract Robotic teleoperation brings great potential for advances within the field of surgery. The ability of a surgeon to reach patient remotely opens exciting opportunities. Early experience with telerobotic surgery has been interesting, but the clinical feasibility remains out of reach, largely due to the deleterious effects of communication delays. Teleoperation tasks are significantly impacted by unavoidable signal latency, which directly results in slower operations, less precision in movements, and increased human errors. Introducing significant changes to the surgical workflow, for example by introducing semi-automation or self-correction, present too significant a technological and ethical burden for commercial surgical robotic systems to adopt. In this paper, we present three simple and intuitive motion scaling solutions to combat teleoperated robotic systems under delay and help improve operator accuracy. Motion scaling offers potentially improved user performance and reduction in errors with minimal change to the underlying teleoperation architecture. To validate the use of motion scaling as a performance enhancer in telesurgery, we conducted a user study with 17 participants, and our results show that the proposed solutions do indeed reduce the error rate when operating under high delay. Florian Richter 0002, Ryan K. Orosco, Michael C. Yip	ICRA	3
2019	Augmented Reality Predictive Displays to Help Mitigate the Effects of Delayed Telesurgery abstract Surgical robots offer the exciting potential for remote telesurgery, but advances are needed to make this technology efficient and accurate to ensure patient safety. Achieving these goals is hindered by the deleterious effects of latency between the remote operator and the bedside robot. Predictive displays have found success in overcoming these effects by giving the operator immediate visual feedback. However, previously developed predictive displays can not be directly applied to telesurgery due to the unique challenges in tracking the 3D geometry of the surgical environment. In this paper, we present the first predictive display for teleoperated surgical robots. The predicted display is stereoscopic, utilizes Augmented Reality (AR) to show the predicted motions alongside the complex tissue found in-situ within surgical environments, and overcomes the challenges in accurately tracking slave-tools in real-time. We call this a Stereoscopic AR Predictive Display (SARPD). To test the SARPD's performance, we conducted a user study with ten participants on the da Vinci® Surgical System. The results showed with statistical significance that using SARPD decreased time to complete task while having no effect on error rates when operating under delay. Florian Richter 0002, Yuheng Zhi, Ryan K. Orosco, Michael C. Yip	ICRA	5
2019	Neural Path Planning: Fixed Time, Near-Optimal Path Generation via Oracle Imitation abstract Fast and efficient path generation is critical for robots operating in complex environments. This motion planning problem is often performed in a robot's actuation or configuration space, where popular pathfinding methods such as A, RRT, get exponentially more computationally expensive to execute as the dimensionality increases or the spaces become more cluttered and complex. On the other hand, if one were to save the entire set of paths connecting all pair of locations in the configuration space a priori, one would run out of memory very quickly. In this work, we introduce a novel way of producing fast and optimal motion plans for static environments by using a stepping neural network approach, called OracleNet. OracleNet uses Recurrent Neural Networks to determine end-to-end trajectories in an iterative manner that implicitly generates optimal motion plans with minimal loss in performance in a compact form. The algorithm is straightforward in implementation while consistently generating near-optimal paths in a single, iterative, end-to-end roll-out. In practice, OracleNet generally has fixed-time execution regardless of the configuration space complexity while outperforming popular pathfinding algorithms in complex environments and higher dimensions1. Mayur J. Bency, Ahmed H. Qureshi, Michael C. Yip	IROS	3
2019	An Open-Source 7-Axis, Robotic Platform to Enable Dexterous Procedures within CT Scanners abstract This paper describes the design, manufacture, and performance of a highly dexterous, low-profile, 7 Degree-of-Freedom (DoF) robotic arm for CT-guided percutaneous needle biopsy. Direct CT guidance allows physicians to localize tumours quickly; however, needle insertion is still performed by hand. This system is mounted to a fully active motion stage superior to the patient's head and teleoperated by a radiologist. Unlike other similar robots, this robot's fully serial-link approach uses a unique combination of belt and cable drives for high-transparency and minimal-backlash, allowing for an expansive working area and numerous approach angles to targets, all while maintaining a small in-bore cross-section of less than 16cm2. Simulations verified the system's expansive collision free work-space and ability to hit targets across the entire chest, as required for lung cancer biopsy. Targeting error was on average <; 1mm on a teleoperated accuracy task, illustrating the system's sufficient accuracy to perform biopsy procedures. While this system was developed for lung biopsies, it can be easily modified for other CT-guided needle based procedures. Additionally, this system is open-hardware. Dimitri A. Schreiber, Daniel B. Shak, Alexander M. Norbash, Michael C. Yip	IROS	4
2019	Model-Free Visual Control for Continuum Robot Manipulators via Orientation Adaptation Mrinal Verghese, Florian Richter 0002, Aaron Gunn, Phil Weissbrod, Michael C. Yip	ISRR	5
2019	Robotic Artificial Muscles: Current Progress and Future Perspectives abstract Robotic artificial muscles are a subset of artificial muscles that are capable of producing biologically inspired motions useful for robot systems, i.e., large power-to-weight ratios, inherent compliance, and large range of motions. These actuators, ranging from shape memory alloys to dielectric elastomers, are increasingly popular for biomimetic robots as they may operate without using complex linkage designs or other cumbersome mechanisms. Recent achievements in fabrication, modeling, and control methods have significantly contributed to their potential utilization in a wide range of applications. However, no survey paper has gone into depth regarding considerations pertaining to their selection, design, and usage in generating biomimetic motions. In this paper, we discuss important characteristics and considerations in the selection, design, and implementation of various prominent and unique robotic artificial muscles for biomimetic robots, and provide perspectives on next-generation muscle-powered robots. Jun Zhang 0025, Jun Sheng, Ciarán T. O'Neill, Conor J. Walsh, Robert J. Wood, Jee-Hwan Ryu, Jaydev P. Desai, Michael C. Yip	IEEE Trans. Robotics	8
2018	Deeply Informed Neural Sampling for Robot Motion Planning abstract Sampling-based Motion Planners (SMPs) have become increasingly popular as they provide collision-free path solutions regardless of obstacle geometry in a given environment. However, their computational complexity increases significantly with the dimensionality of the motion planning problem. Adaptive sampling is one of the ways to speed up SMPs by sampling a particular region of a configuration space that is more likely to contain an optimal path solution. Although there are a wide variety of algorithms for adaptive sampling, they rely on hand-crafted heuristics; furthermore, their performance decreases significantly in high-dimensional spaces. In this paper, we present a neural network-based adaptive sampler for motion planning called Deep Sampling-based Motion Planner (DeepSMP). DeepSMP generates samples for SMPs and enhances their overall speed significantly while exhibiting efficient scalability to higher-dimensional problems. DeepSMP's neural architecture comprises of a Contractive AutoEncoder which encodes given workspaces directly from a raw point cloud data, and a Dropout-based stochastic deep feedforward neural network which takes the workspace encoding, start and goal configuration, and iteratively generates feasible samples for SMPs to compute end-to-end collision-free optimal paths. DeepSMP is not only consistently computationally efficient in all tested environments but has also shown remarkable generalization to completely unseen environments. We evaluate DeepSMP on multiple planning problems including planning of a point-mass robot, rigid-body, 6-link robotic manipulator in various 2D and 3D environments. The results show that on average our method is at least 7 times faster in point-mass and rigid-body case and about 28 times faster in 6-link robot case than the existing state-of-the-art. Ahmed H. Qureshi, Michael C. Yip	IROS	2
2017	On the Control and Properties of Supercoiled Polymer Artificial Muscles abstract Robotics have long sought an actuation technology comparable to or as capable as biological muscle tissue. Natural muscles exhibit a high power-to-weight ratio, inherent compliance and damping, fast action, and a high dynamic range. They also produce joint displacements and forces without the need for gearing or additional hardware. Recently, supercoiled commercially available polymer threads (sewing thread or nylon fishing lines) have been used to create significant mechanical power in a muscle-like form factor. Heating and cooling the polymer threads causes contraction and expansion, which can be utilized for actuation. In this paper, we describe the working principle of supercoiled polymer (SCP) actuation and explore the controllability and properties of these threads. We show that under appropriate environmental conditions, the threads are suitable as a building block for a controllable artificial muscle. We leverage off-the-shelf silver-coated threads to enable rapid electrical heating while the low thermal mass allows for rapid cooling. We utilize both thermal and thermomechanical models for feed-forward and feedback control. The resulting SCP actuator regulates to desired force levels in as little as 28 ms. Together with its inherent stiffness and damping, this is sufficient for a position controller to execute large step movements in under 100 ms. This controllability, high performance, the mechanical properties, and the extremely low material cost are indicative of a viable artificial muscle. Michael C. Yip, Günter Niemeyer	IEEE Trans. Robotics	1
2015	High-performance robotic muscles from conductive nylon sewing thread abstract Natural muscles exhibit high power-to-weight ratios, inherent compliance and damping, fast actuation and high dynamic ranges. Unfortunately, traditional robotic actuators have been unable to attain similar properties, especially in a slender muscle-like form factor. Recently, super-coiled polymer (SCP) actuators have rejuvenated the promise of an artificial muscle. Constructed from commercial nylon fishing line or sewing thread and twisted until coils form, these lightweight actuators have been shown to produce significant mechanical power when thermally cycled. In this paper, we develop a thermomechanical and thermoelectric model of SCP actuators, and examine their controllability. With off-the-shelf conductive sewing thread, we show the ability to produce controlled forces in under 30 ms, exceeding human muscle performance. Finally, we use SCP actuators in a robotic hand to demonstrate their applicability as a low-cost, high performance robotic muscle. Michael C. Yip, Günter Niemeyer	ICRA	1
2015	WoodenHaptics: A Starting Kit for Crafting Force-Reflecting Spatial Haptic Devices abstract Spatial haptic interfaces have been around for 20 years. Yet, few affordable devices have been produced, and the design space in terms of physical workspace and haptic fidelity of devices that have been produced are limited and discrete. In this paper, an open-source, open-hardware module-based kit is presented that allows an interaction designer with little electro-mechanical experience to manufacture and assemble a fully working spatial haptic interface. It also allows for modification in shape and size as well as tuning of parameters to fit a particular task or application. Results from an evaluation showed that the haptic quality of the WoodenHaptics device was on par with a Phantom Desktop and that a novice could assemble it with guidance in a normal office space. This open source starting kit, uploaded free-to-download online, affords sketching in hardware; it unsticks the hardware from being a highly-specialized and esoteric craft to being an accessible and user-friendly technology, while maintaining the feel of high-fidelity haptics. Jonas Forsslund, Michael C. Yip, Eva-Lotta Sallnäs	TEI	2
2014	Model-Less Feedback Control of Continuum Manipulators in Constrained Environments abstract Continuum manipulators offer a means for robot manipulation in a constrained environment, where the manipulator body can safely interact with, comply with, and navigate around obstacles. However, obstacle interactions impose constraints that conform the robot body into arbitrary shapes regardless of actuator positions. Generally, these effects cannot be wholly sensed on a continuum manipulator and, therefore, render model-based controllers incorrect, leading to artificial singularities and unstable behavior. We present a task-space closed-loop controller for continuum manipulators that does not rely on a model and can be used in constrained environments. Using an optimal control strategy on a tendon-driven robot, we demonstrate this method, which we term model-less control, which allows the manipulator to interact with several constrained environments in a stable manner. To the best of our knowledge, this is the first work in controlling continuum manipulators without using a model. Michael C. Yip, David B. Camarillo	IEEE Trans. Robotics	1
2012	Registration of 3D Ultrasound Through an Air-Tissue Boundary abstract In this study we evaluated a new method for registering three-dimensional ultrasound (3DUS) data to external coordinate systems. First, 3DUS was registered to the stereo endoscope of a da Vinci Surgical System by placing a registration tool against an air-tissue boundary so that the 3DUS could image ultrasound fiducials while the stereo endoscope could image camera markers on the same tool. The common points were used to solve the registration between the 3DUS and camera coordinate systems. The target registration error (TRE) when imaging through a PVC tissue phantom ranged from 3.85 1.76 mm to 1.82 1.03 mm using one to four registration tool positions. TRE when imaging through an ex-vivo liver tissue sample ranged from 2.36 1.01 mm to 1.51 0.70 mm using one to four registration tool positions. Second, using a similar method, 3DUS was registered to the kinematic coordinate system of a da Vinci Surgical System by using the da Vinci surgical manipulators to identify common points on an air-tissue boundary. TRE when imaging through a PVC tissue phantom was 0.95 0.38 mm. This registration method is simpler and potentially more accurate than methods using commercial motion tracking systems. This method may be useful in the future in augmented reality systems for laparoscopic and robotic-assisted surgery. Troy K. Adebar, Michael C. Yip, Tim Salcudean, Robert Rohling, Christopher Y. Nguan, Larry Goldenberg	IEEE Trans. Medical Imaging	2
2012	Tissue Tracking and Registration for Image-Guided Surgery abstract Vision-based tracking of tissue is a key component to enable augmented reality during a surgical operation. Conven- tional tracking techniques in computer vision rely on identifying strong edge features or distinctive textures in a well-lit environ- ment; however endoscopic tissue images do not have strong edge features, are poorly lit and exhibit a high degree of specular reflection. Therefore, prior work in achieving densely populated 3D features for describing tissue surface profiles require complex image processing techniques and have been limited in providing stable, long-term tracking or real-time processing. In this paper, we present an integrated framework for ac- curately tracking tissue in surgical stereo-cameras at real-time speeds. We use a combination of the STAR feature detector and Binary Robust Independent Elementary Features to acquire salient features that can be persistently tracked at high frame rates. The features are then used to acquire a densely-populated map of the deformations of tissue surface in 3D. We evaluate the method against popular feature algorithms in in-vivo animal study video sequences, and we also apply the proposed method to human partial nephrectomy video sequences. We extend the salient feature framework to support region tracking in order to maintain the spatial correspondence of a tracked region of tissue or a medical image registration to the surrounding tissue. In-vitro tissue studies show registration accuracies of 1.3-3.3 mm using a rigid-body transformation method. Michael C. Yip, David G. Lowe, Tim Salcudean, Robert Rohling, Christopher Y. Nguan	IEEE Trans. Medical Imaging	1
2010	Performance analysis of a manipulation task in time-delayed teleoperation abstract There is ample research on the stabilization of haptic teleoperation systems under communication time delay. Little attention, however, has so far been paid to the usefulness of delayed haptic feedback on task performance. While the usefulness of haptic feedback in no-delay teleoperation has been previously established, this paper investigates whether haptic feedback helps to improve task performance in the presence of delay. We consider peg-in-the-hole insertion, which is a dexterous manipulation task requiring high force levels at certain points during the task execution. Through a user study involving unilateral and bilateral teleoperation experiments under different delays, it is observed that in both unilateral and bilateral teleoperation, the task completion time increases as delay increases. It is also seen that haptic feedback helps reduce the amount and rate of energy transfer to the environment and the occurrence of larger robot/environment interaction forces. However, with the users mindful of minimizing contact forces, haptic feedback causes the task to take more time compared to no haptic feedback regardless of the time delay. Thus, for tasks where low completion times are crucial given a tolerance for larger forces, unilateral feedback may be sufficient. Michael C. Yip, Mahdi Tavakoli, Robert D. Howe	IROS	1
2010	3D Ultrasound to Stereoscopic Camera Registration through an Air-Tissue Boundary Michael C. Yip, Troy K. Adebar, Robert Rohling, Tim Salcudean, Christopher Y. Nguan	MICCAI (2)	1
2009	Robotic Force Stabilization for Beating Heart Intracardiac Surgery Shelten G. Yuen, Michael C. Yip, Nikolay V. Vasilyev, Douglas P. Perrin, Pedro J. del Nido, Robert D. Howe	MICCAI (1)	2