We present a comparative study for discriminative anatomy detection in high dimensional neuroimaging data. While most studies solve this problem using mass univariate approaches, recent works show better accuracy and variable selection using a sparse classification model. Two types of image-based regularization methods have been proposed in the literature based on either a Graph Net (GN) model or a total variation (TV) model. These studies showed increased classification accuracy and interpretability of results when using image-based regularization, but did not look at the accuracy and quality of the recovered significant regions. In this paper, we theoretically prove bounds on the recovered sparse coefficients and the corresponding selected image regions in four models (two based on GN penalty and two based on TV penalty). Practically, we confirm the theoretical findings by measuring the accuracy of selected regions compared with ground truth on simulated data. We also evaluate the stability of recovered regions over cross-validation folds using real MRI data. Our findings show that the TV penalty is superior to the GN model. In addition, we showed that adding an l2 penalty improves the accuracy of estimated coefficients and selected significant regions for the both types of models.

The proportions of muscle and fat tissues in the human body, referred to as body composition is a vital measurement for cancer patients. Body composition has been recently linked to patient survival and the onset/recurrence of several types of cancers in numerous cancer research studies. This paper introduces a fully automatic framework for the segmentation of muscle and fat tissues from CT images to estimate body composition. We developed a novel finite element method (FEM) deformable model that incorporates a priori shape information via a statistical deformation model (SDM) within the template-based segmentation framework. The proposed method was validated on 1000 abdominal and 530 thoracic CT images and we obtained very good segmentation results with Jaccard scores in excess of 90% for both the muscle and fat regions.

Existing scene flow approaches mainly focus on two-frame stereo-pair configurations and reconstruct an image-based representation of scene flow. Instead, we propose a variational formulation of scene flow relative to a coarse proxy geometry, which is better suited for many views. Furthermore, a linear basis is used to represent temporal surface flow, allowing for longer-range temporal correspondence with fewer variables. Our formulation takes known proxy motion into account (e.g, if the proxy is a tracked human subject), which enables 3D trajectory reconstruction when only a single view is available. Additionally, through the appropriate proxy and basis, our framework generalizes existing approaches for scene flow, optic-flow, and two-frame stereo. We illustrate results on real-data for both static and moving proxy surfaces over several frames.

To tele-operate a robot, visual feedback is critical. However, communication channel latency can delay feedback to the point where the operator is impeded in performing his task. This work presents a vision-based “predictive display” system that compensates for visual delay. The approach is online and relatively uncalibrated, thus it has the advantage of being useful in unknown environments and many applications. From monocular eye-in-hand video, we incrementally compute a 3D graphics model of the robot site in real time using our new technique. The method exploits free-space/occlusion constraints on the scene to produce a physically consistent mesh. Novel vantage points are immediately rendered in response to the operator's control commands, without waiting for delayed video. We implement a full prototype tele-operation system where the operator controls, via a PHANTOM Omni device, a Barrett WAM robot mounted on a mobile Segway. Experiments with this setup validate the efficacy of the proposed approach. We demonstrate significant improvement in task completion time with predictive display on a real robot, while our previous related results were established only in simulation.

We propose a novel approach for improving level set segmentation methods by embedding the potential functions from a discriminatively trained conditional random field (CRF) into a level set energy function. The CRF terms can be efficiently estimated and lead to both discriminative local potentials and edge regularizers that take into account interactions among the labels. Unlike discrete CRFs, the use of a continuous level set framework allows the natural use of flexible continuous regularizers such as shape priors. We show promising experimental results for the method on two difficult medical image segmentation tasks.

Tumor segmentation from MRI data is an important but time consuming task performed manually by medical experts. Automating this process is challenging due to the high diversity in appearance of tumor tissue among different patients and, in many cases, similarity between tumor and normal tissue. We propose a semi-automatic interactive brain tumor segmentation system that incorporates 2D interactive and 3D automatic tools with the ability to adjust operator control. The provided methods are based on an energy that incorporates region statistics computed on available MRI modalities and the usual regularization term. The energy is efficiently minimized on-line using graph cut. Experiments with radiation oncologists testing the semi-automatic tool vs. a manual tool show that the proposed system improves both segmentation time and repeatability.

Tumor segmentation from MRI data is an important but time consuming task performed manually by medical experts. Automating this process is challenging due to the high diversity in appearance of tumor tissue, among different patients and, in many cases, similarity between tumor and normal tissue. One other challenge is how to make use of prior information about the appearance of normal brain. In this paper we propose a variational brain tumor segmentation algorithm that extends current approaches from texture segmentation by using a high dimensional feature set calculated from MRI data and registered atlases. Using manually segmented data we learn a statistical model for tumor and normal tissue. We show that using a conditional model to discriminate between normal and abnormal regions significantly improves the segmentation results compared to traditional generative models. Validation is performed by testing the method on several cancer patient MRI scans.

Delays in the visual feedback can seriously impact operator performance in telerobotics. In predictive display synthesized visual feedback is rendered immediately in response to operator motions. In this paper we present a system using a geometric and appearance model that is captured using structure-from-motion by an uncalibrated camera. The geometric model is integrated into a registration based tracking algorithm that allows stable tracking of full 3D pose of the robot. Experimentally we show that predictive scene views can be rendered with both high visual fidelity and metric accuracy.

In this paper, we present an image-based robot incremental localization algorithm which uses a panoramic image-based map enhanced with depth from a laser range finder. The image-based map (model) contains both intensity information as well as sparse 3D geometric features. By assuming motion continuity, a robot can use the depth information in the image-model to project the relevant 3D model features, specifically vertical lines, of the environment to its camera coordinate frame. To determine its location, the robot first acquires an intensity image and then matches the 2D geometric features in the image with the projected model features. The first contribution of this research is that we avoid the difficult problem of full 3D reconstruction from images by employing a range sensor registered with respect to the intensity image sensor; secondly, we provide an algorithm that performs incremental robot localization using only 2D images. Experimental results in indoor map building and localization demonstrate the feasibility of our approach and evaluate the performance of the algorithm.

In tele-robotics delayed visual feedback to the human operator can degrade task performance significantly. To improve this, predictive display, uses a scene model to estimate and render immediate visual feedback based on the operator's control commands. Traditional predictive display involves the calibration and overlay of an a-priori model with the delayed real video feedback. In this paper we present an image-based method where the scene geometry and appearance is captured using structure-from-motion by an uncalibrated eye-in-hand camera mounted on the remote robot. The model is then compressed and transmitted to the operator site, where it is used to generate immediate feedback in response to the operators movements. Calibration problems are avoided since the model is captured by the same scene camera as is being simulated in the predictive display. We show experiments where we capture the appearance of a robot hand and transmit it over the network to the operator site where the model renders scene appearance change in response to operator viewpoint motion.

A long-standing goal in image-based modeling and rendering is to capture a scene from camera images and construct a sufficient model to allow photo-realistic rendering of new views. With the confluence of computer graphics and vision, the combination of research on recovering geometric structure from un-calibrated cameras with modeling and rendering has yielded numerous new methods. Yet, many challenging issues remain to be addressed before a sufficiently general and robust system could be built to (for instance) allow an average user to model their home and garden from camcorder video. This tutorial aims to give researchers and students in computer graphics a working knowledge of relevant theory and techniques covering the steps from real-time vision for tracking and the capture of scene geometry and appearance, to the efficient representation and real-time rendering of image-based models. It also includes hands-on demos of real-time visual tracking, modeling and rendering systems.

We present a method that using only an uncalibrated camera allows the capture of object geometry and appearance, and then at a later stage registration and AR overlay into a new scene. Using only image information first a coarse object geometry is obtained using structure-from-motion, then a dynamic, view dependent texture is estimated to account for the differences between the reprojected coarse model and the training images. In AR rendering, the object structure is interactively aligned in one frame by the user, object and scene structure is registered, and rendered in subsequent frames by a virtual scene camera, with parameters estimated from real-time visual tracking. Using the same viewing geometry for both object acquisition, registration, and rendering ensures consistency and minimizes errors.

We present and contrast four methods for registering 3D range data to 2D images. Two are calibration techniques that recover the rigid transformation between the sensor poses, based on point or line correspondences. The two others recover a direct, image based mapping between the data sets. The accuracy of each method is experimentally evaluated on test patterns and objects. We found that the point based calibration method is the best approach to recover a global registration between the two sensors, while an image-based method performed best when registering local regions.

The problem of capturing real world scenes and then accurately rendering them is particularly difficult for fine-scale 3D structure. Similarly, it is difficult to capture, model and animate non-rigid motion. We present a method where small image changes are captured as a time varying (dynamic) texture. In particular, a coarse geometry is obtained from a sample set of images using structure from motion. This geometry is then used to subdivide the scene and to extract approximately stabilized texture patches. The residual statistical variability in the texture patches is captured using a PCA basis of spatial filters. The filters coefficients are parameterized in camera pose and object motion. To render new poses and motions, new texture patches are synthesized by modulating the texture basis. The texture is then warped back onto the coarse geometry. We demonstrate how the texture modulation and projective homography-based warps can be achieved in real-time using hardware accelerated OpenGL. Experiments comparing dynamic texture modulation to standard texturing are presented for objects with complex geometry (a flower) and non-rigid motion (human arm motion capturing the non-rigidities in the joints, and creasing of the shirt). Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Image Based Rendering

Mobile robots must be able to learn, use and maintain models of their environments. Two fundamental and interrelated problems in mobile robotics are mapping and localization. We present here a new type of image-based map formed by panoramic models enriched with depth and 3D planarity information. We take an advantage of the scene geometry implicitly contained in the model to localize a mobile robot that is moving in the same environment. The novelty of this model compared to the existing image-based maps is that the motion of the robot is not restricted to a predefined path or locations close to the original images. Experimental results demonstrate this approach.