Surgical tool pose estimation has been proven to be useful for high- and low- level feedback tasks including safety-enhancement, semantic feedback and surgical skill assessment. Tool pose estimation using monocular camera input is a well-studied research problem as the monocular camera is one of the ubiquitous sensor across the spectrum of robotic devices. Current state-of-the art methods for visual tool pose estimation are computationally expensive and require elaborate geometric and appearance models of surgical tools. We propose a visual tool pose estimation method that maps the visual bounding box to the 3D tool pose without any explicit knowledge of tool geometry using Gaussian process regression. The proposed approach can be generalized to any surgical tool and provides tool pose estimates with a variance estimate in real-time. We demonstrate rigorous evaluation of the method under various conditions that might effect the estimation process. In order to evaluate the algorithm, we have instrumented a standard box trainer kit with two laparoscopic tools to get simultaneous ground truth pose and a video feed.

HD Video from the (monocular or binocular) endoscopic camera provides a rich real-time sensing channel from surgical site to the surgeon console in various Minimally Invasive Surgery (MIS) procedures. However, a real-time framework for video understanding would be critical for tapping into the rich information-content provided by the non-invasive and well-established digital endoscopic video-streaming modality. While contemporary research focuses on enhancing aspects such as tool-tracking within the challenging visual scenes, we consider the associated problem of using that rich (but often compromised) streaming visual data to discover the underlying semantic attributes of the tools. Directly analyzing the surgical videos to extract more realistic attributes online can aid in the decision-making and feedback aspects. We propose a novel probabilistic attribute labelling framework with Bayesian filtering to identify associated semantics (open/closed, stained with blood etc.) to ultimately give semantic feedback to the surgeon. Our robust video-understanding framework overcomes many of the challenges (tissue deformations, image specularities, clutter, tool-occlusion due to blood and/or organs) under realistic in-vivo surgical conditions. Specifically, this manuscript performs rigorous experimental analysis of the resulting method with varying parameters and different visual features on a data-corpus consisting of real surgical procedures performed on patients with da Vinci Surgical System [9].

Variability in motor rehabilitation program outcomes can be attributed not only to individual components (human patient/rehabilitation equipment) but also to their system-level interactions. Thus, effective deployment of a rehabilitation program depends upon: 1) suitable therapist selection of user-device ergonomics; 2) adjustable device settings; and 3) exercise regimen parameters; to achieve desired system-level motor performance. In this paper, we discuss aspects of creation of a virtual design environment, leveraging tools from musculoskeletal analysis, optimization, and simulation-based design, to permit therapists to rapidly evaluate and systematically customize rehabilitation programs. Specifically, this framework is intended to facilitate 1) parametric study of ergonomic/device/regimen settings on musculoskeletal performance; 2) use of design tools such as optimization for decision support in arriving at the best program; and 3) scaffolded examination of linkage between form and function by iterative what-if type of analyses. We use two case studies (bicep-curling and motor rehabilitative driving) to highlight benefits of such simulation-based rehabilitation program evaluation.