Bronchoscopy is currently the least invasive method for definitively diagnosing lung cancer, which kills more people in the United States than any other form of cancer. Successfully diagnosing suspicious lung nodules requires accurate localization of the bronchoscope relative to a planned biopsy site in the airways. This task is challenging because the lung deforms intraoperatively due to respiratory motion, the airways lack photometric features, and the anatomy's appearance is repetitive. In this paper, we introduce a real-time camera-based method for accurately localizing a bronchoscope with respect to a planned needle insertion pose. Our approach uses deep learning and accounts for deformations and overcomes limitations of global pose estimation by estimating pose relative to anatomical landmarks. Specifically, our learned model considers airway bifurcations along the airway wall as landmarks because they are distinct geometric features that do not vary significantly with respiratory motion. We evaluate our method in a simulated dataset of lungs undergoing respiratory motion. The results show that our method generalizes across patients and localizes the bronchoscope with accuracy sufficient to access the smallest clinically-relevant nodules across all levels of respiratory deformation, even in challenging distal airways. Our method could enable physicians to perform more accurate biopsies and serve as a key building block toward accurate autonomous robotic bronchoscopy.

Deep learning-based, single-view depth estimation methods have recently shown highly promising results. However, such methods ignore one of the most important features for determining depth in the human vision system, which is motion. We propose a learning-based, multi-view dense depth map and odometry estimation method that uses Recurrent Neural Networks (RNN) and trains utilizing multi-view image reprojection and forward-backward flow-consistency losses. Our model can be trained in a supervised or even unsupervised mode. It is designed for depth and visual odometry estimation from video where the input frames are temporally correlated. However, it also generalizes to single-view depth estimation. Our method produces superior results to the state-of-the-art approaches for single-view and multi-view learning-based depth estimation on the KITTI driving dataset.

Large-scale deformations of a tubular object, or generalized cylinder, are often defined by a target shape for its center curve, typically using a parametric target curve. This task is non-trivial for free-form deformations or direct manipulation methods because it is hard to manually control the centerline by adjusting control points. Most skeleton-based methods are no better, again due to the small number of manually adjusted control points. In this paper, we propose a method to deform a generalized cylinder based on its skeleton composed of a centerline and orthogonal cross sections. Although we are not the first to use such a skeleton, we propose a novel skeletonization method that tries to minimize the number of intersections between neighboring cross sections by means of a relative curvature condition to detect intersections. The mesh deformation is first defined geometrically by deforming the centerline and mapping the cross sections. Rotation minimizing frames are used during mapping to control twisting. Secondly, given displacements on the cross sections, the deformation is decomposed into finely subdivided regions. We limit distortion at these vertices by minimizing an elastic thin shell bending energy, in linear time. Our method can handle complicated generalized cylinders such as the human colon.

We present a novel approach for improving the shape statistics of medical image objects by generating correspondence of skeletal points. Each object's interior is modeled by an s-rep, i.e., by a sampled, folded, two-sided skeletal sheet with spoke vectors proceeding from the skeletal sheet to the boundary. The skeleton is divided into three parts: the up side, the down side, and the fold curve. The spokes on each part are treated separately and, using spoke interpolation, are shifted along that skeleton in each training sample so as to tighten the probability distribution on those spokes' geometric properties while sampling the object interior regularly. As with the surface/boundary-based correspondence method of Cates et al., entropy is used to measure both the probability distribution tightness and the sampling regularity, here of the spokes' geometric properties. Evaluation on synthetic and real world lateral ventricle and hippocampus data sets demonstrate improvement in the performance of statistics using the resulting probability distributions. This improvement is greater than that achieved by an entropy-based correspondence method on the boundary points.

We present a scheme that propagates a reference skeletal model (s-rep) into a particular case of an object, thereby propagating the initial shape-related layout of the skeleton-to-boundary vectors, called spokes. The scheme represents the surfaces of the template as well as the target objects by spherical harmonics and computes a warp between these via a thin plate spline. To form the propagated s-rep, it applies the warp to the spokes of the template s-rep and then statistically refines. This automatic approach promises to make s-rep fitting robust for complicated objects, which allows s-rep based statistics to be available to all. The improvement in fitting and statistics is significant compared with the previous methods and in statistics compared with a state-of-the-art boundary based method.

In image-guided radiotherapy (IGRT) of disease sites subject to respiratory motion, soft tissue deformations can affect localization accuracy. We describe the application of a method of 2D/3D deformable registration to soft tissue localization in abdomen. The method, called registration efficiency and accuracy through learning a metric on shape (REALMS), is designed to support real-time IGRT. In a previously developed version of REALMS, the method interpolated 3D deformation parameters for any credible deformation in a deformation space using a single globally-trained Riemannian metric for each parameter. We propose a refinement of the method in which the metric is trained over a particular region of the deformation space, such that interpolation accuracy within that region is improved. We report on the application of the proposed algorithm to IGRT in abdominal disease sites, which is more challenging than in lung because of low intensity contrast and nonrespiratory deformation. We introduce a rigid translation vector to compensate for nonrespiratory deformation, and design a special region-of-interest around fiducial markers implanted near the tumor to produce a more reliable registration. Both synthetic data and actual data tests on abdominal datasets show that the localized approach achieves more accurate 2D/3D deformable registration than the global approach.

Abstract In this paper, we introduce a new texture metamorphosis approach for interpolating texture samples from a source texture into a target texture. We use a new energy optimization scheme derived from optimal control principles which exploits the structure of the metamorphosis optimality conditions. Our approach considers the change in pixel position and pixel appearance in a single framework. In contrast to previous techniques that compute a global warping based on feature masks of textures, our approach allows to transform one texture into another by considering both intensity values and structural features of textures simultaneously. We demonstrate the usefulness of our approach for different textures, such as stochastic, semi‐structural and regular textures, with different levels of complexities. Our method produces visually appealing transformation sequences with no user interaction.

One goal of statistical shape analysis is the discrimination between two populations of objects. Whereas traditional shape analysis was mostly concerned with single objects, analysis of multi-object complexes presents new challenges related to alignment and pose. In this paper, we present a methodology for discriminant analysis of multiple objects represented by sampled medial manifolds. Non-euclidean metrics that describe geodesic distances between sets of sampled representations are used for alignment and discrimination. Our choice of discriminant method is the distance-weighted discriminant because of its generalization ability in high-dimensional, low sample size settings. Using an unbiased, soft discrimination score, we associate a statistical hypothesis test with the discrimination results. We explore the effectiveness of different choices of features as input to the discriminant analysis, using measures like volume, pose, shape, and the combination of pose and shape. Our method is applied to a longitudinal pediatric autism study with 10 subcortical brain structures in a population of 70 subjects. It is shown that the choices of type of global alignment and of intrinsic versus extrinsic shape features, the latter being sensitive to relative pose, are crucial factors for group discrimination and also for explaining the nature of shape change in terms of the application domain.

An important goal of statistical shape analysis is the discrimination between populations of objects, exploring group differences in morphology not explained by standard volumetric analysis. Certain applications additionally require analysis of objects in their embedding context by joint statistical analysis of sets of interrelated objects. In this paper, we present a framework for discriminant analysis of populations of 3-D multi-object sets. In view of the driving medical applications, a skeletal object parametrization of shape is chosen since it naturally encodes thickening, bending and twisting. In a multi-object setting, we not only consider a joint analysis of sets of shapes but also must take into account differences in pose. Statistics on features of medial descriptions and pose parameters, which include rotational frames and distances, uses a Riemannian symmetric space instead of the standard Euclidean metric. Our choice of discriminant method is the distance weighted discriminant (DWD) because of its generalization ability in high dimensional, low sample size settings. Joint analysis of 10 subcortical brain structures in a pediatric autism study demonstrates that multi-object analysis of shape results in a better group discrimination than pose, and that the combination of pose and shape performs better than shape alone. Finally, given a discriminating axis of shape and pose, we can visualize the differences between the populations.

Two major factors preventing the routine clinical use of finite-element analysis for image registration are: 1) the substantial labor required to construct a finite-element model for an individual patient's anatomy and 2) the difficulty of determining an appropriate set of finite-element boundary conditions. This paper addresses these issues by presenting algorithms that automatically generate a high quality hexahedral finite-element mesh and automatically calculate boundary conditions for an imaged patient. Medial shape models called m-reps are used to facilitate these tasks and reduce the effort required to apply finite-element analysis to image registration. Encouraging results are presented for the registration of CT image pairs which exhibit deformation caused by pressure from an endorectal imaging probe and deformation due to swelling.

A primary goal of statistical shape analysis is to describe the variability of a population of geometric objects. A standard technique for computing such descriptions is principal component analysis. However, principal component analysis is limited in that it only works for data lying in a Euclidean vector space. While this is certainly sufficient for geometric models that are parameterized by a set of landmarks or a dense collection of boundary points, it does not handle more complex representations of shape. We have been developing representations of geometry based on the medial axis description or m-rep. While the medial representation provides a rich language for variability in terms of bending, twisting, and widening, the medial parameters are not elements of a Euclidean vector space. They are in fact elements of a nonlinear Riemannian symmetric space. In this paper, we develop the method of principal geodesic analysis, a generalization of principal component analysis to the manifold setting. We demonstrate its use in describing the variability of medially-defined anatomical objects. Results of applying this framework on a population of hippocampi in a schizophrenia study are presented.

Medial representation of a three-dimensional (3-D) object or an ensemble of 3-D objects involves capturing the object interior as a locus of medial atoms, each atom being two vectors of equal length joined at the tail at the medial point. Medial representation has a variety of beneficial properties, among the most important of which are 1) its inherent geometry, provides an object-intrinsic coordinate system and thus provides correspondence between instances of the object in and near the object(s); 2) it captures the object interior and is, thus, very suitable for deformation; and 3) it provides the basis for an intuitive object-based multiscale sequence leading to efficiency of segmentation algorithms and trainability of statistical characterizations with limited training sets. As a result of these properties, medial representation is particularly suitable for the following image analysis tasks; how each operates will be described and will be illustrated by results: segmentation of objects and object complexes via deformable models; segmentation of tubular trees, e.g., of blood vessels, by following height ridges of measures of fit of medial atoms to target images; object-based image registration via medial loci of such blood vessel trees; statistical characterization of shape differences between control and pathological classes of structures. These analysis tasks are made possible by a new form of medial representation called m-reps, which is described.

The clinical recognition of abnormal vascular tortuosity, or excessive bending, twisting, and winding, is important to the diagnosis of many diseases. Automated detection and quantitation of abnormal vascular tortuosity from three-dimensional (3-D) medical image data would, therefore, be of value. However, previous research has centered primarily upon two-dimensional (2-D) analysis of the special subset of vessels whose paths are normally close to straight. This report provides the first 3-D tortuosity analysis of clusters of vessels within the normally tortuous intracerebral circulation. We define three different clinical patterns of abnormal tortuosity. We extend into 3-D two tortuosity metrics previously reported as useful in analyzing 2-D images and describe a new metric that incorporates counts of minima of total curvature. We extract vessels from MRA data, map corresponding anatomical regions between sets of normal patients and patients with known pathology, and evaluate the three tortuosity metrics for ability to detect each type of abnormality within the region of interest. We conclude that the new tortuosity metric appears to be the most effective in detecting several types of abnormalities. However, one of the other metrics, based on a sum of curvature magnitudes, may be more effective in recognizing tightly coiled, "corkscrew" vessels associated with malignant tumors.

The period of the Medical Image Display and Analysis Group (MIDAG) so far is 1974-2002: more than 27 years. We began with a focus on two-dimensional (2-D) display: contrast enhancement, display scale choice, and display device standardization. We co-invented adaptive histogram equalization and later improved it to contrast-limited AHE, and we were perhaps the first to show that adaptive contrast enhancement, i.e., care in the mapping between recorded and displayed intensity and variation of that mapping with the local properties of the image, could significantly affect diagnostic or therapeutic decisions. MIDAG prides itself in having affected medical practice and, thus, the lives of patients. Despite the fact that bringing research from conception to actual medical use is a process sometimes taking a decade, the largest fraction, perhaps all, of our graduate students and faculty are attracted to these applications of computers by this altruism. Areas in which MIDAG research has come to this fruition are the uses of color display in nuclear medicine, the standardization of CRT display and the realization of how many bits of intensity are needed, and the use of tested contrast enhancement methods in areas of medical image use where subtle changes must be detected. Medical areas where we have had an effect are mammography, a major target area for both the standardization and contrast enhancement ends, and portal imaging in radiotherapy, a target area for contrast enhancement. In the 1980s, some of MIDAG's attention moved to image analysis. Also beginning in the 1980s we began to make some contributions to the notions of scale space description of images. With emphasis on the development of segmentation by deformable models and our aforementioned principle that validation is a critical part of research developing image analysis and display methods, we have begun to seriously face the issues of how to validate segmentation and how to choose the parameters of a segmentation method. Our experimental design and analysis techniques involve a variety of new methods for repeated variables designs.

This paper presents a multiscale framework based on a medial representation for the segmentation and shape characterization of anatomical objects in medical imagery. The segmentation procedure is based on a Bayesian deformable templates methodology in which the prior information about the geometry and shape of anatomical objects is incorporated via the construction of exemplary templates. The anatomical variability is accommodated in the Bayesian framework by defining probabilistic transformations on these templates. The transformations, thus, defined are parameterized directly in terms of natural shape operations, such as growth and bending, and their locations. A preliminary validation study of the segmentation procedure is presented. We also present a novel statistical shape analysis approach based on the medial descriptions that examines shape via separate intuitive categories, such as global variability at the coarse scale and localized variability at the fine scale. We show that the method can be used to statistically describe shape variability in intuitive terms such as growing and bending.

A model of object shape by nets of medial and boundary primitives is justified as richly capturing multiple aspects of shape and yet requiring representation space and image analysis work proportional to the number of primitives. Metrics are described that compute an object representation's prior probability of local geometry by reflecting variabilities in the net's node and link parameter values, and that compute a likelihood function measuring the degree of match of an image to that object representation. A paradigm for image analysis of deforming such a model to optimize a posteriori probability is described, and this paradigm is shown to be usable as a uniform approach for object definition, object-based registration between images of the same or different imaging modalities, and measurement of shape variation of an abnormal anatomical object, compared with a normal anatomical object. Examples of applications of these methods in radiotherapy, surgery, and psychiatry are given.

A method is proposed for the automatic, rapid, and stable identification and measurement of objects in three-dimensional (3-D) images. It is based on local shape properties derived statistically from populations of medial primitives sought throughout the image space. These shape properties are measured at medial locations within the object and include scale, orientation, endness, and medial dimensionality. Medial dimensionality is a local shape property differentiating sphere-like, cylinder-like, and slab-like structures, with intermediate dimensionality also possible. Endness is a property found at the cap of a cylinder or the edge of a slab. In terms of an application, the cardiac left ventricle (LV) during systole is modeled as a large dark cylinder with an apical cap, terminated at the other end by a thin bright slab-like mitral valve (MV). Such a model, containing medial shape properties at just a few locations, along with the relative distances and orientations between these locations, is intuitive and robust and permits automated detection of the LV axis in vivo, using real-time 3-D (RT3D) echocardiography. The statistical nature of these shape properties allows their extraction, even in the presence of noise, and permits statistical geometric measurements without exact delineation of boundaries, as demonstrated in determining the volume of balloons in RT3D scans. The inherent high speed of the method is appropriate for real-time clinical use.

Transparency can be a useful device for depicting multiple overlapping surfaces in a single image. The challenge is to render the transparent surfaces in such a way that their 3D shape can be readily understood and their depth distance from underlying structures clearly perceived. This paper describes our investigations into the use of sparsely-distributed discrete, opaque texture as an artistic device for more explicitly indicating the relative depth of a transparent surface and for communicating the essential features of its 3D shape in an intuitively meaningful and minimally occluding way. The driving application for this work is the visualization of layered surfaces in radiation therapy treatment planning data, and the technique is illustrated on transparent isointensity surfaces of radiation dose. We describe the perceptual motivation and artistic inspiration for defining a stroke texture that is locally oriented in the direction of greatest normal curvature (and in which individual strokes are of a length proportional to the magnitude of the curvature in the direction they indicate), and we discuss two alternative methods for applying this texture to isointensity surfaces defined in a volume. We propose an experimental paradigm for objectively measuring observers' ability to judge the shape and depth of a layered transparent surface, in the course of a task which is relevant to the needs of radiotherapy treatment planning, and use this paradigm to evaluate the practical effectiveness of our approach through a controlled observer experiment based on images generated from actual clinical data.

Transparency can be a useful device for simultaneously depicting multiple superimposed layers of information in a single image. However, in computer-generated pictures-as in photographs and in directly viewed actual objects-it can often be difficult to adequately perceive the three-dimensional shape of a layered transparent surface or its relative depth distance from underlying structures. Inspired by artists' use of line to show shape, we have explored methods for automatically defining a distributed set of opaque surface markings that intend to portray the three-dimensional shape and relative depth of a smoothly curving layered transparent surface in an intuitively meaningful (and minimally occluding) way. This paper describes the perceptual motivation, artistic inspiration and practical implementation of an algorithm for "texturing" a transparent surface with uniformly distributed opaque short strokes, locally oriented in the direction of greatest normal curvature, and of length proportional to the magnitude of the surface curvature in the stroke direction. The driving application for this work is the visualization of layered surfaces in radiation therapy treatment planning data, and the technique is illustrated on transparent isointensity surfaces of radiation dose.

There are many applications that can benefit from the simultaneous display of multiple layers of data. The objective in these cases is to render the layered surfaces in a such way that the outer structures can be seen and seen through at the same time. The paper focuses on the particular application of radiation therapy treatment planning, in which physicians need to understand the three dimensional distribution of radiation dose in the context of patient anatomy. We describe a promising technique for communicating the shape and position of the transparent skin surface while at the same time minimally occluding underlying isointensity dose surfaces and anatomical objects: adding a sparse, opaque texture comprised of a small set of carefully chosen lines. We explain the perceptual motivation for explicitly drawing ridge and valley curves on a transparent surface, describe straightforward mathematical techniques for detecting and rendering these lines, and propose a small number of reasonably effective methods for selectively emphasizing the most perceptually relevant lines in the display.

The techniques of a posteriori image restoration and iterative image feature extraction are described and compared. Image feature extraction methods known as graduated nonconvexity (GNC); variable conductance diffusion (VCD), anisotropic diffusion, and biased anisotropic diffusion (BAD), which extract edges from noisy images, are compared with a restoration/feature extraction method known as mean field annealing (MFA). All are shown to be performing the same basic operation: image relaxation. This equivalence shows the relationship between energy minimization methods and spatial analysis methods and between their respective parameters of temperature and scale. As a result of the equivalence, VCD is demonstrated to minimize a cost function, and that cost is specified explicitly. Furthermore, operations over scale space are shown to be a method of avoiding local minima.>

Two methods for identifying and analyzing the multiresolution behavior of ridges and valleys in grey-scale images are described. The first method uses the tools of differential geometry to focus on local image behavior. The resulting vertex curves mark the tops of ridges and bottoms of valleys in an image. The second method focuses on the global drainage patterns of rainfall on a terrain map. The resulting watershed boundaries also identify the tops of ridges and bottoms of valleys in an image. By following these two geometric representations through scale space, the authors build resolution hierarchies on ridges and valleys in the image that can be utilized for interactive image segmentation.>

The authors present the intensity axis of symmetry (IAS) method for describing the shape of structures in grey-scale images. They describe the spatial and intensity variations of the image simultaneously rather than by the usual two-step process of using intensity properties of the image to segment an image into regions and describing the spatial shape of these regions. The result is an image shape description that is useful for a number of computer vision applications. The method relies on minimizing an active surface functional that provides coherence in both the spatial and intensity dimensions while deforming into an axis of symmetry. Shape-based image segmentation is possible by identifying image regions associated with individual components of the IAS. The resulting image regions have geometric coherence and correspond well to visually meaningful objects in medical images.>

The authors have achieved rates as high as 15 frames per second for interactive direct visualization of 3D data by trading some function for speed, while volume rendering with a full complement of ramp classification capabilities is performed at 1.4 frames per second. These speeds have made the combination of region selection with volume rendering practical for the first time. Semantic-driven selection, rather than geometric clipping, has proved to be a natural means of interacting with 3D data. Internal organs in medical data or other regions of interest can be built from preprocessed region primitives. The resulting combined system has been applied to real 3D medical data with encouraging results.>

A computer algorithm which segments gray-scale images into regions of interest (objects) has been developed. These regions can provide the basis for scene analysis (including shape-parameter calculation) or surface-based, shaded-graphics display. The algorithm creates a tree structure for image description by defining a linking relationship between pixels in successively blurred versions of the initial image. The image is described in terms of nested light and dark regions. This algorithm, successfully implemented in one, two, and three dimensions, can theoretically work with any number of dimensions. The interactive postprocessing developed technique selects regions from the descriptive tree for display in several ways: pointing to a branch of the image description tree, specifying by sliders the range of scale and/or intensity of all regions which should be displayed, and pointing (on the original image) to any pixel in the desired region. The algorithm has been applied to approximately 15 computer tomography (CT) images of the abdomen.>

A fundamental approach for providing an image description in terms of visually sensible image regions is described. It involves a) the representation of the image by a that captures image information and then b) the definition of a hierarchy of components of that by the order of annihilation of those components as the image is continuously simplified by lowering the scale. The information-capturing essential structure is chosen so that image regions are associated with each component during the image simplification. To guarantee image simplification, successive Gaussian blurring is chosen as the means of scale lowering. We argue that an that describes shape in both the spatial and intensity dimensions will produce an image description most likely to be useful for computer or human specification of image objects. In particular, we suggest that the intensity axis of symmetry (IAS) satisfies all desirable criteria for an structure. With such shape-based structures the approach of image description via annihilation under image simplification becomes a very attractive paradigm.

A method is proposed that produces a shape description in the form of a hierarchy by scale of simple symmetric axis sequences. An axis segment that is a child of another has smaller scale and is seen as a branch of its parent. The scale value and parent-child relationship are induced by following the symmetric axis under successive reduction of resolution. The result, in two or three dimensions, is a figure¿rather than boundary¿oriented shape description that has natural segments and is insensitive to noise in the object description. The general approach of hierarchy production by following a feature through successive resolution reduction will be presented, as will methods of resolution reduction and computer implementation. Also, the relation of this figure-based shape description to those based on boundary curvature will be briefly discussed.

Blum's two-dimensional shape description method based on the symmetric axis transform (SAT) is generalized to three dimensions. The method uniquely decomposes an object into a collection of sub-objects each drawn from three separate, but not completely independent, primitive sets defined in the paper: width primitives, based on radius function properties; axis primitives, based on symmetric axis curvatures; and boundary primitives, based on boundary surface curvatures. Width primitives are themselves comprised of two components: slope districts and curvature districts. Visualizing the radius function as if it were the height function of some mountainous terrain, each slope district corresponds to a mountain face together with the valley below it. Curvature districts further partition each slope district into regions that are locally convex, concave, or saddle-like. Similarly, axis (boundary) primitives are regions of the symmetric surface where the symmetric surface (boundary surfaces) are locally convex, concave, or saddle-like. Relations among the primitive sets are discussed.