PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center video dataset. In this work we investigated the generalizability of phase recognition algorithms in a multicenter setting including more difficult recognition tasks such as surgical action and surgical skill. METHODS: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 h was created. Labels included framewise annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 international Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 research teams trained and submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. RESULTS: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n = 9 teams), for instrument presence detection between 38.5% and 63.8% (n = 8 teams), but for action recognition only between 21.8% and 23.3% (n = 5 teams). The average absolute error for skill assessment was 0.78 (n = 1 team). CONCLUSION: Surgical workflow and skill analysis are promising technologies to support the surgical team, but there is still room for improvement, as shown by our comparison of machine learning algorithms. This novel HeiChole benchmark can be used for comparable evaluation and validation of future work. In future studies, it is of utmost importance to create more open, high-quality datasets in order to allow the development of artificial intelligence and cognitive robotics in surgery.

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

A novel approach for accurate markerless motion capture combining a precise tracking algorithm with a database of articulated models is presented. The tracking approach employs an articulated iterative closest point algorithm with soft-joint constraints for tracking body segments in visual hull sequences. The database of articulated models is derived from a combination of human shapes and anthropometric data, contains a large variety of models and closely mimics variations found in the human population. The database provides articulated models that closely match the outer appearance of the visual hulls, e.g. matches overall height and volume. This information is paired with a kinematic chain enhanced through anthropometric regression equations. Deviations in the kinematic chain from true joint center locations are compensated by the soft-joint constraints approach. As a result accurate and a more anatomical correct outcome is obtained suitable for biomechanical and clinical applications. Joint kinematics obtained using this approach closely matched joint kinematics obtained from a marker based motion capture system.

Abstract A method is proposed for photo‐realistic modeling and visualization of a growing tree. Recent visualization methods have focused on producing smoothly blending branching structures, however, these methods fail to account for the inclusion of non‐smooth features such as branch bark ridges and bud scale scars. These features constitute an important visual aspect of a living tree, and are also observed to vary over time. The proposed method incorporates these features by using an hierarchical implicit modeling system, which provides a variety of tools for combining surface components in both smooth and non smooth configurations. A procedural interface to this system supports the use of inverse modeling, which is a global‐to‐local methodology, where the local properties of plant organs are described in terms of their global position within the tree architecture. Inverse modeling is used to define both the tree structure at any time, and a continuous developmental sequence for the tree from a seedling. These techniques provide an intuitive paradigm for the definition of complex tree growth sequences and their subsequent visualization, based solely on observed phenomena. Thus, a key advantage is that they do not require any knowledge of, or simulation of, the underlying biological processes. Categories and Subject Descriptors (according to ACM CCS): I.3.5 [Computer Graphics]: Curve, surface, solid, and object representations I.3.7 [Computer Graphics]: Animation

In contrast to the extensively researched modeling of plant architecture, the modeling of plant organs largely remains an open problem. We propose a method for modeling lobed leaves. This method extends the concept of sweeps to branched skeletons. The input of the model is a 2D leaf silhouette, which can be defined interactively or derived from a scanned leaf image. The algorithm computes the skeleton (medial axis) of the leaf and approximates it using spline curves interconnected into a branching structure (sticky splines). The leaf surface is then constructed by sweeping a generating curve along these splines. The orientation of the generating curve is adjusted to properly capture the shape of the leaf blade near the extremities and branching points of the skeleton, and to avoid self-intersections of the surface. The leaf model can be interactively modified by editing the shape of the silhouette and the skeleton. It can be further manipulated in 3D using functions that control turning, bending, and twisting of each lobe.

We integrate into plant models three elements of plant representation identified as important by artists: posture (manifested in curved stems and elongated leaves), gradual variation of features, and the progression of the drawing process from overall silhouette to local details. The resulting algorithms increase the visual realism of plant models by offering an intuitive control over plant form and supporting an interactive modeling process. The algorithms are united by the concept of expressing local attributes of plant architecture as functions of their location along the stems.