Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci® Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset.

This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. daVinciNet performs concurrent end-effector trajectory and surgical state predictions using features extracted from multiple data streams, including robot kinematics, endoscopic vision, and system events. We evaluate our proposed model on an extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci® Xi surgical system and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our model achieves up to 93.85% short-term (0.5s) and 82.11% long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error.

We present an algorithm to solve a sequential stochastic decision making problem whereby a robot is subject to multiple objective functions and is asked to complete a number of subgoals specified using a subset of linear temporal logic. Each subgoal is associated with a desired satisfaction probability that will be met in expectation by the policy produced by the algorithm. Our method relies on the theory of constrained Markov Decision Processes and on methods coming from the realm of formal verification. The key idea is the definition of a product operation that can recursively incorporate more and more subgoals into the underlying planner. Ultimately, a policy is computed solving a linear program and we outline conditions for the existence and correctness of the solution. Our findings are validated in various simulation scenarios.

Constrained Markov Decision Processes offer a principled way to tackle sequential decision problems with multiple objectives. Although they could be very valuable in numerous robotic applications, to date their use has been quite limited. One of the reasons is that their solution requires to solve constrained linear programs with a large number of variables and this is computationally demanding, especially when considering dynamic environments. In this paper we propose a hierarchical approach to solve large CMDPs. States are clustered into macro states and relevant parameters like transition probabilities and costs are extracted with a Monte Carlo approach. Macro states are created with the objective of grouping together states with similar costs while preserving feasibility. We illustrate the value of our findings in a path planning scenario where the robot moves through an environment characterized by different risk levels. Our approach largely outperforms the non-hierarchical method and we also show how it prevails over methods based on fixed partitioning strategies.