2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin.

Discrete tomography problems require the generation of an underlying attenuation field over a combinatorial set that renders the direct application of conventional tomographic reconstruction techniques obsolete. In this work, we propose a solution approach based on a new variable splitting coupled with the alternating direction method of multipliers (ADMM). The new approach creates subproblems that can be solved using existing and fast techniques, such as graph-cut methods, and results in overall solutions of excellent quality. In particular, compared to state-of-the art techniques, we obtain reliable representations of the underlying fields with substantially lower angular coverage. We develop the method under both Gaussian and Poisson noise models and show results with both binary and multilabel phantoms.

In recent years, baggage screening at airports has included the use of dual-energy X-ray computed tomography (DECT), an advanced technology for nondestructive evaluation. The main challenge remains to reliably find and identify threat objects in the bag from DECT data. This task is particularly hard due to the wide variety of objects, the high clutter, and the presence of metal, which causes streaks and shading in the scanner images. Image noise and artifacts are generally much more severe than in medical CT and can lead to splitting of objects and inaccurate object labeling. The conventional approach performs object segmentation and material identification in two decoupled processes. Dual-energy information is typically not used for the segmentation, and object localization is not explicitly used to stabilize the material parameter estimates. We propose a novel learning-based framework for joint segmentation and identification of objects directly from volumetric DECT images, which is robust to streaks, noise and variability due to clutter. We focus on segmenting and identifying a small set of objects of interest with characteristics that are learned from training images, and consider everything else as background. We include data weighting to mitigate metal artifacts and incorporate an object boundary field to reduce object splitting. The overall formulation is posed as a multilabel discrete optimization problem and solved using an efficient graph-cut algorithm. We test the method on real data and show its potential for producing accurate labels of the objects of interest without splits in the presence of metal and clutter.

Efficient graph-cut methods have been used with great success for labeling and denoising problems occurring in computer vision. Unfortunately, the presence of linear image mappings has prevented the use of these techniques in most discrete-amplitude image reconstruction problems. In this paper, we develop a graph-cut based framework for the direct solution of discrete amplitude linear image reconstruction problems cast as regularized energy function minimizations. We first analyze the structure of discrete linear inverse problem cost functions to show that the obstacle to the application of graph-cut methods to their solution is the variable mixing caused by the presence of the linear sensing operator. We then propose to use a surrogate energy functional that overcomes the challenges imposed by the sensing operator yet can be utilized efficiently in existing graph-cut frameworks. We use this surrogate energy functional to devise a monotonic iterative algorithm for the solution of discrete valued inverse problems. We first provide experiments using local convolutional operators and show the robustness of the proposed technique to noise and stability to changes in regularization parameter. Then we focus on nonlocal, tomographic examples where we consider limited-angle data problems. We compare our technique with state-of-the-art discrete and continuous image reconstruction techniques. Experiments show that the proposed method outperforms state-of-the-art techniques in challenging scenarios involving discrete valued unknowns.

The rapid decrease in the dimensions of integrated circuits has necessitated corresponding higher-resolution methods for defect imaging. Current state of the art, defect imaging systems are reaching the limits of their resolution. In this work, we are proposing a new overcomplete dictionary based sparse signal imaging framework to improve the resolution and localization in confocal microscopy systems for backside optical integrated circuit defect imaging. The domain of integrated circuit imaging is particularly suitable for the application of overcomplete dictionaries in an image reconstruction framework because the images are highly structured, containing predictable building blocks derivable from the corresponding computer aided design layouts. This structure provides a strong and natural a-priori dictionary for scene reconstruction. This dictionary prior is coupled with a physically-based observation model to create enhanced scene reconstructions. The approach is described and results on simulated data are provided.

In this paper we propose a graph cut method for solving a multi-label linear inverse problem with an arbitrary system matrix. Graph cuts are efficient methods for solving pixel-labeling and early vision problems. Energy function minimization problems that occur in image denoising are easily solved by graph cut techniques. However, applying graph cuts to inverse problems which have a non-diagonal system matrix becomes challenging, as a data cost of one pixel depends on intensities of other pixels. Such cost functions are not graph representable. In this paper, we propose an iterative method for minimization of energy functions occurring in inverse problems, where a graph-representable Taylor approximation of the original cost function is rapidly solved via a graph cut method at each iteration.