Organ shape reconstruction based on a single-projection image during treatment has wide clinical scope, e.g., in image-guided radiotherapy and surgical guidance. We propose an image-to-graph convolutional network that achieves deformable registration of a three-dimensional (3D) organ mesh for a low-contrast two-dimensional (2D) projection image. This framework enables simultaneous training of two types of transformation: from the 2D projection image to a displacement map, and from the sampled per-vertex feature to a 3D displacement that satisfies the geometrical constraint of the mesh structure. Assuming application to radiation therapy, the 2D/3D deformable registration performance is verified for multiple abdominal organs that have not been targeted to date, i.e., the liver, stomach, duodenum, and kidney, and for pancreatic cancer. The experimental results show shape prediction considering relationships among multiple organs can be used to predict respiratory motion and deformation from digitally reconstructed radiographs with clinically acceptable accuracy.

The positions of nodules can change because of intraoperative lung deflation, and the modeling of pneumothorax-associated deformation remains a challenging issue for intraoperative tumor localization. In this study, we introduce spatial and geometric analysis methods for inflated/deflated lungs and discuss heterogeneity in pneumothorax-associated lung deformation. Contrast-enhanced CT images simulating intraoperative conditions were acquired from live Beagle dogs. The images contain the overall shape of the lungs, including all lobes and internal bronchial structures, and were analyzed to provide a statistical deformation model that could be used as prior knowledge to predict pneumothorax. To address the difficulties of mapping pneumothorax CT images with topological changes and CT intensity shifts, we designed deformable mesh registration techniques for mixed data structures including the lobe surfaces and the bronchial centerlines. Three global-to-local registration steps were performed under the constraint that the deformation was spatially continuous and smooth, while matching visible bronchial tree structures as much as possible. The developed framework achieved stable registration with a Hausdorff distance of less than 1 mm and a target registration error of less than 5 mm, and visualized deformation fields that demonstrate per-lobe contractions and rotations with high variability between subjects. The deformation analysis results show that the strain of lung parenchyma was 35% higher than that of bronchi, and that deformation in the deflated lung is heterogeneous.

Respiratory motion and the associated deformations of abdominal organs and tumors are essential information in clinical applications. However, inter- and intra-patient multi-organ deformations are complex and have not been statistically formulated, whereas single organ deformations have been widely studied. In this paper, we introduce a multi-organ deformation library and its application to deformation reconstruction based on the shape features of multiple abdominal organs. Statistical multi-organ motion/deformation models of the stomach, liver, left and right kidneys, and duodenum were generated by shape matching their region labels defined on four-dimensional computed tomography images. A total of 250 volumes were measured from 25 pancreatic cancer patients. This paper also proposes a per-region-based deformation learning using the non-linear kernel model to predict the displacement of pancreatic cancer for adaptive radiotherapy. The experimental results show that the proposed concept estimates deformations better than general per-patient-based learning models and achieves a clinically acceptable estimation error with a mean distance of 1.2 ± 0.7 mm and a Hausdorff distance of 4.2 ± 2.3 mm throughout the respiratory motion.

Interactive image segmentation that uses a bounding box containing the foreground has gained great popularity because of its convenience. However, its performance is often degraded when the bounding box is not tight enough or covers large background regions. To solve this problem, this letter proposes a novel segmentation algorithm called “SuperCut”. This algorithm provides robust segmentation in one cut even with loose bounding boxes. In the proposed SuperCut, superpixels of the whole image are first computed, and then a background similarity map for the superpixels inside the bounding box is created based on Haar-Wavelet feature and pixel intensities. Finally, a max-flow cut is made based on the map. The background similarity map reduces the influence from possible background regions inside the bounding box, which leads to the efficient segmentation of SuperCut. To evaluate the effectiveness of the proposed SuperCut, we compared it with some recent published interactive image segmentation techniques. The experimental results show that the proposed SuperCut outperforms other compared methods especially when using loose bounding boxes.

This paper introduces a concept of endoscopic image augmentation that overlays shape changes to support cutting procedures. This framework handles the history of measured drill tip's location as a volume label, and visualizes the remains to be cut overlaid on the endoscopic image in real time. We performed a cutting experiment, and the efficacy of the cutting aid was evaluated among shape similarity, total moved distance of a cutting tool, and the required cutting time. The results of the experiments showed that cutting performance was significantly improved by the proposed framework.

This study proposes a method to estimate elastic deformation using silhouettes obtained from multiple endoscopic images. Our method can estimate the intraoperative deformation of organs using a volumetric mesh model reconstructed from preoperative CT data. We use this elastic body silhouette information of elastic bodies not to model the shape but to estimate the local displacements. The model shape is updated to satisfy the silhouette constraint while preserving the shape as much as possible. The result of the experiments showed that the proposed methods could estimate the deformation with root mean square (RMS) errors of 5.0–10 mm.

In preoperative planning for mandibular reconstructive surgery, it is necessary to determine the osteotomy lines for fibular shaping and the proper placement of fibular segments in the mandible. Although virtual surgical planning has been utilized in preoperative decision making, current software designs require manual operation and a trial-and-error process to refine the reconstruction plan. We have developed volumetric fibular transfer simulation software that can quickly design a preoperative plan based on direct volume manipulation and quantitative comparison with the patient's original mandible. We propose three quantitative shape indicators-volume ratio, contour error, and maximum projection-for symmetrical lesions of the mandible, and have implemented a parallel computation algorithm for the semiautomatic placement of fibular segments. Using this virtual planning software, we conducted a retrospective study of the computed tomography data from nine patients. We found that combining direct volume manipulation with real-time local search of placement improved the applicability of the planning system to optimize mandibular reconstruction.

BACKGROUND: Recent advances in microscopy enable the acquisition of large numbers of tomographic images from living tissues. Three-dimensional microscope images are often displayed with volume rendering by adjusting the transfer functions. However, because the emissions from fluorescent materials and the optical properties based on point spread functions affect the imaging results, the intensity value can differ locally, even in the same structure. Further, images obtained from brain tissues contain a variety of neural structures such as dendrites and axons with complex crossings and overlapping linear structures. In these cases, the transfer functions previously used fail to optimize image generation, making it difficult to explore the connectivity of these tissues. RESULTS: This paper proposes an interactive visual exploration method by which the transfer functions are modified locally and interactively based on multidimensional features in the images. A direct editing interface is also provided to specify both the target region and structures with characteristic features, where all manual operations can be performed on the rendered image. This method is demonstrated using two-photon microscope images acquired from living mice, and is shown to be an effective method for interactive visual exploration of overlapping similar structures. CONCLUSIONS: An interactive visualization method was introduced for local improvement of visualization by volume rendering in two-photon microscope images containing regions in which linear nerve structures crisscross in a complex manner. The proposed method is characterized by the localized multidimensional transfer function and interface where the parameters can be determined by the user to suit their particular visualization requirements.

This paper proposes a 3-D cardiovascular modeling system based on neonatal echocardiographic images. With the system, medical doctors can interactively construct patient-specific cardiovascular models, and share the complex topology and the shape information. For the construction of cardiovascular models with a variety of congenital heart diseases, we propose a set of algorithms and interface that enable editing of the topology and shape of the 3-D models. In order to facilitate interactivity, the centerline and radius of the vessels are used to edit the surface of the heart vessels. This forms a skeleton where the centerlines of blood vessel serve as the nodes and edges, while the radius of the blood vessel is given as an attribute value to each node. Moreover, parent-child relationships are given to each skeleton. They are expressed as the directed acyclic graph, where the skeletons are viewed as graph nodes and the connecting points are graph edges. The cardiovascular models generated from some patient data confirmed that the developed technique is capable of constructing cardiovascular disease models in a tolerable timeframe. It is successful in representing the important structures of the patient-specific heart vessels for better understanding in preoperative planning and electric medical recording of the congenital heart disease.

This paper introduces a new design to allow interactive, direct manipulation of volume data on volumetrically rendered images. We present an adaptive volume proxy mesh which serves not to define surfaces, but to encode the geometry and physical state of the volume. This system performs a modeling-free form of direct volume deformation by adaptively constructing the whole geometric structure on the background while keeping the structure transparent to the user. A complex, high-frequency structure is locally encoded into the mesh on-the-fly based on the user's volume of interest. This paper also presents a scheme for volume shading/shadowing applied in combination with our framework for improving reality in the visualization of time-varying deformable objects. We demonstrate examples of geometry-encoded direct volume manipulation and application to volume exploration and surgical simulation.

This paper presents a new set of techniques by which surgeons can interactively manipulate patient-specific volumetric models for sharing surgical process. To handle physical interaction between the surgical tools and organs, we propose a simple surface-constraint-based manipulation algorithm to consistently simulate common surgical manipulations such as grasping, holding and retraction. Our computation model is capable of simulating soft-tissue deformation and incision in real time. We also present visualization techniques in order to rapidly visualize time-varying, volumetric information on the deformed image. This paper demonstrates the success of the proposed methods in enabling the simulation of surgical processes, and the ways in which this simulation facilitates preoperative planning and rehearsal.

This paper proposes vertex-preserving cutting methods on finite element models for interactive soft tissue simulation. Unlike existing methods, we aim to shape variety of incisions using only initial vertices of tetrahedral meshes. Neither tetrahedral decomposition nor vertex creation is used. The number of vertices is preserved. This avoids increase of computation cost as well as allows fast update of physical status of finite element models. To preserve 3D shape and sharp feature of initial meshes through on-the-fly mesh modification, constraints are introduced to the topological update scheme. In our model, the size of stiffness matrix is constant. Our framework efficiently simulates several varieties of smooth incisions with sufficient quality for surgical simulation, and also achieves interactive performance in complex meshes with thousands of elements.