Current research strive has tremendously changed the horizon of computer vision tasks in multiple agents tracking. Nevertheless, in the research of robotic assisted surgery, reliable surgical instrument tracking imposes challenge due to the high complexity in state modeling for the hierarchical structure of the instrument versus de-coupling the spatial-temporal correlations naturally embedded in the task. In this paper, we present a new tracking paradigm integrating the trajectory prediction to reduce the data association error that is propagated from the false detection. As a key component in the system, a proposed predictor disentangles the hierarchical modeling and agent kinematic learning by introducing inductive attention mechanism in spatial-temporal graph network. Experiments on real anatomical datasets show that our tracking-by-prediction scheme improves overall localization accuracy over the frames by up to 81%, in comparison to the generic pipelines of tracking, even with transductive graph representation learning, with a large margin of gain in terms of precise localization.

OBJECTIVE: Robotic-assisted minimally invasive surgery (RAMIS) became a common practice in modern medicine and is widely studied. Surgical procedures require prolonged and complex movements; therefore, classifying surgical gestures could be helpful to characterize surgeon performance. The public release of the JIGSAWS dataset facilitates the development of classification algorithms; however, it is not known how algorithms trained on dry-lab data generalize to real surgical situations. METHODS: We trained a Long Short-Term Memory (LSTM) network for the classification of dry lab and clinical-like data into gestures. RESULTS: We show that a network that was trained on the JIGSAWS data does not generalize well to other dry-lab data and to clinical-like data. Using rotation augmentation improves performance on dry-lab tasks, but fails to improve the performance on clinical-like data. However, using the same network architecture, adding the six joint angles of the patient-side manipulators (PSMs) features, and training the network on the clinical-like data together lead to notable improvement in the classification of the clinical-like data. DISCUSSION: Using the JIGSAWS dataset alone is insufficient for training a gesture classification network for clinical data. However, it can be very informative for determining the architecture of the network, and with training on a small sample of clinical data, can lead to acceptable classification performance. SIGNIFICANCE: Developing efficient algorithms for gesture classification in clinical surgical data is expected to advance understanding of surgeon sensorimotor control in RAMIS, the automation of surgical skill evaluation, and the automation of surgery.

Knowledge of interaction forces during teleoperated robot-assisted surgery could be used to enable force feedback to users and evaluate tissue handling skill. However, direct force sensing at the end-effector is challenging because it requires biocompatible, sterilizable, and cost-effective sensors. Vision-based neural networks are a promising approach for providing useful force estimates, though questions remain about generalization to new scenarios and real-time inference. We present a force estimation neural network that uses RGB images and robot state as inputs. Using a self-collected dataset, we compared the network to variants that included only a single input type, and evaluated how they generalized to new viewpoints, workspace positions, materials, and tools. We found that the vision-only network was sensitive to shifts in viewpoints, while networks with state inputs were sensitive to vertical shifts in workspace. The network with both state and vision inputs had the highest accuracy for an unseen tool, while the state-only network was most accurate for an unseen material. Through feature removal studies, we found that using only force features produced better accuracy than using only kinematic features as input. The network with both state and vision inputs outperformed a physics-based model in accuracy for seen material. It showed comparable accuracy but faster computation times than a recurrent neural network, making it better suited for real-time applications.

Robotic-Assisted Minimally Invasive Surgery allows for easy recording of kinematic data, and presents excellent opportunities for data-intensive approaches to assessment of surgical skill, system design, and automation of procedures. However, typical surgical cases result in long data streams, and therefore, automated segmentation into gestures is important. The public release of the JIGSAWS dataset allowed for developing and benchmarking data-intensive segmentation algorithms. However, this dataset is small and the gestures are similar in their structure and directions. This may limit the generalization of the algorithms to real surgical data that are characterized by movements in arbitrary directions. In this paper, we use a recurrent neural network to segment a suturing task, and demonstrate one such generalization problem-limited generalization to rotation. We propose a simple augmentation that can solve this problem without collecting new data, and demonstrate its benefit using: (1) the JIGSAWS dataset, and (2) a new dataset that we recorded with a da Vinci Research Kit. Our study highlights the prospect of using data augmentation in the analysis of kinematic data in surgical data science.