Diffusion models have found phenomenal success as expressive priors for solving inverse problems, but their extension beyond natural images to more structured scientific domains remains limited. Motivated by applications in materials science, we aim to reduce the number of measurements required from an expensive imaging modality of interest, by leveraging side information from an auxiliary modality that is much cheaper to obtain. To deal with the non-differentiable and black-box nature of the forward model, we propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information, requiring significantly less amount of data from the expensive microscopy modality.

Automatic tracking of large-scale crowded targets are of particular importance in many applications, such as crowded people/vehicle tracking in video surveillance, fiber tracking in materials science, and cell tracking in biomedical imaging. This problem becomes very challenging when the targets show similar appearance and the interslice/ inter-frame continuity is low due to sparse sampling, camera motion and target occlusion. The main challenge comes from the step of association which aims at matching the predictions and the observations of the multiple targets. In this paper we propose a new groupwise method to explore the target group information and employ the within-group correlations for association and tracking. In particular, the within-group association is modeled by a nonrigid 2D Thin-Plate transform and a sequence of group shrinking, group growing and group merging operations are then developed to refine the composition of each group. We apply the proposed method to track large-scale fibers from microscopy material images and compare its performance against several other multi-target tracking methods. We also apply the proposed method to track crowded people from videos with poor inter-frame continuity.

Computed tomography is increasingly enabling scientists to study physical processes of materials at micron scales. The MBIR framework provides a powerful method for CT reconstruction by incorporating both a measurement model and prior model. Classically, the choice of prior has been limited to models enforcing local similarity in the image data. In some material science problems, however, much more may be known about the underlying physical process being imaged. Moreover, recent work in Plug-And-Play decoupling of the MBIR problem has enabled researchers to look beyond classical prior models, and innovations in methods of data acquisition such as interlaced view sampling have also shown promise for imaging of dynamic physical processes. In this paper, we propose an MBIR framework with a physics based prior model - namely the Cahn-Hilliard equation. The Cahn-Hilliard equation can be used to describe the spatiotemporal evolution of binary alloys. After formulating the MBIR cost with Cahn-Hilliard prior, we use Plug-And-Play algorithm with ICD optimization to minimize this cost. We apply this method to simulated data using the interlaced-view sampling method of data acquisition. Results show superior reconstruction quality compared to the Filtered Back Projection. Though we use Cahn-Hilliard equation as one instance, the method can be easily extended to use any other physics-based prior model for a different set of applications.

Fast and accurate characterization of fiber micro-structures plays a central role for material scientists to analyze physical properties of continuous fiber reinforced composite materials. In materials science, this is usually achieved by continuously cross-sectioning a 3D material sample for a sequence of 2D microscopic images, followed by a fiber detection/tracking algorithm through the obtained image sequence. To speed up this process and be able to handle larger size material samples, this paper proposes sparse sampling with larger inter-slice distance in cross sectioning and develops a new algorithm that can robustly track large-scale fibers from such a sparsely sampled image sequence. In particular, the problem is formulated as multi-target tracking, and the Kalman filters are applied to track each fiber along the image sequence. One main challenge in this tracking process is to correctly associate each fiber to its observation given that: fiber observations are of large scale, crowded, and show very similar appearances in a 2D slice and there may be a large gap between the predicted location of a fiber and its observation in the sparse sampling. To address this challenge, a novel group-wise association algorithm is developed by leveraging the fact that fibers are implanted in bundles and the fibers in the same bundle are highly correlated through the image sequence. In experiments, the proposed algorithm is tested on three tiles of 100-slice S200 material samples and the tracking performance is evaluated using 1136 human annotated ground-truth fiber tracks. Both quantitative and qualitative results show that the proposed algorithm clearly outperforms the state-of-the-art multiple-target tracking algorithms on sparsely sampled image sequences.

We propose a coercive approach to simultaneously register and segment multi-modal images which share similar spatial structure. Registration is done at the region level to facilitate data fusion while avoiding the need for interpolation. The algorithm performs alternating minimization of an objective function informed by statistical models for pixel values in different modalities. Hypothesis tests are developed to determine whether to refine segmentations by splitting regions. We demonstrate that our approach has significantly better performance than the state-of-the-art registration and segmentation methods on microscopy images.

Many samples imaged in structural biology and material science contain several similar particles at random locations and orientations. Model-based iterative reconstruction (MBIR) methods can in principle be used to exploit such redundancies in images through log prior probabilities that accurately account for non-local similarity between the particles. However, determining such a log prior term can be challenging. Several denoising algorithms like non-local means (NLM) successfully capture such non-local redundancies, but the problem is two-fold: NLM is not explicitly formulated as a cost function, and neither can it capture similarity between randomly oriented particles. In this paper, we propose a rotationally-invariant nonlocal means (RINLM) algorithm, and describe a method to implement RINLM as a prior model using a novel framework that we call plug-and-play priors. We introduce the idea of patch pre-rotation to make RINLM computationally tractable. Finally, we showcase image denoising and 2D tomography results, using the proposed RINLM algorithm, as we highlight high reconstruction quality, image sharpness, and artifact suppression.

Enforcing a specific topology in image segmentation is a very important but challenging problem, which has attracted much attention in the computer vision community. Most recent works on topology-constrained image segmentation focus on binary segmentation, where the topology is often described by the connectivity of both foreground and background. In this paper, we develop a new multi-labeling method to enforce topology in multi-label image segmentation. In this case, we not only require each segment to be a connected region (intra-segment topology), but also require specific adjacency relations between each pair of segments (inter-segment topology). We develop our method in the context of segmentation propagation, where a segmented template image defines the topology, and our goal is to propagate the segmentation to a target image while preserving the topology. Our method requires good spatial structure continuity between the template and the target such that the template segmentation can be used as a good initialization for segmenting the target. In addition, we focus on multi-label segmentation where a segment and its adjacent segments form a ring structure, which is among the most complex type of inter-segment topology for 2D structures. We apply the proposed method to segment 3D metallic image volumes for the underlying grain structures and achieve better results than several comparison methods. Finally, we also apply the proposed method to interactive segmentation and stereo matching applications.

This letter considers statistical estimation problems where the probability distribution of the observed random variable is invariant with respect to actions of a finite topological group. It is shown that any such distribution must satisfy a restricted finite mixture representation. When specialized to the case of distributions over the sphere that are invariant to the actions of a finite spherical symmetry group G, a group-invariant extension of the Von Mises Fisher (VMF) distribution is obtained. The G-invariant VMF is parameterized by location and scale parameters that specify the distribution's mean orientation and its concentration about the mean, respectively. Using the restricted finite mixture representation these parameters can be estimated using an Expectation Maximization (EM) maximum likelihood (ML) estimation algorithm. This is illustrated for the problem of mean crystal orientation estimation under the spherically symmetric group associated with the crystal form, e.g., cubic or octahedral or hexahedral. Simulations and experiments establish the advantages of the extended VMF EM-ML estimator for data acquired by Electron Backscatter Diffraction (EBSD) microscopy of a polycrystalline Nickel alloy sample.

Synchrotron based X-ray tomography is widely used for three dimensional imaging of materials at the micron scale. Tomographic data collected from a synchrotron is often affected by non-idealities in the measurement system and sudden “blinding” of detector pixels during the acquisition. Typically, reconstructions are done using analytical reconstruction techniques combined with pre/post-processing steps to correct for the non-idealities, resulting in loss of detail while still producing noisy reconstructions with some artifacts. In this paper, we present a model-based iterative reconstruction (MBIR) algorithm for synchrotron X-ray tomography that can automatically handle the non-idealities as a part of the reconstruction. First, we develop a forward model that accounts for the non-idealities in the measurement system and for the occurrence of outliers in the measurement. Next, we combine the forward model with a prior model of the object to formulate the MBIR cost function and propose an algorithm to minimize the cost. Results on a real data set show that the MBIR reconstructions are superior to the analytical reconstructions effectively suppressing noise as well as other artifacts.

The Markov Random Field (MRF) has been used extensively in Image Processing as a means of smoothing interfaces between differing regions in an image. The MRF applies a total boundary length `energy' penalty that is subsequently minimized by an inversion algorithm. The minimization of energy implies a force associated with boundaries, the sum of which must equal zero at every point at equilibrium. This requirement leads to long range interactions, resulting from the short-range interactions of the MRF, which biases segmentation results. This work uses a simple Bayesian MRF regularized segmentation method to show that classical results from Surface Science are reproduced when segmenting regions of low contrast. This has implications, both in the Materials Science and Image Processing fields.

We propose a segmentation and anomaly detection method for electron backscatter diffraction (EBSD) images. In contrast to conventional methods that require Euler angles to be extracted from diffraction patterns, the proposed method operates on the patterns directly. We use a forward model implemented as a dictionary of diffraction patterns generated by a detailed physics-based simulation of EBSD. The combination of full diffraction patterns and a dictionary allows anomalies to be detected at the same time as grains are segmented, and also increases robustness to noise and instrument blur. The proposed method is demonstrated on a sample of the Ni-base alloy IN100.

High angle annular dark field (HAADF)-scanning transmission electron microscope (STEM) data is increasingly being used in the physical sciences to research materials in 3D because it reduces the effects of Bragg diffraction seen in bright field TEM data. Typically, tomographic reconstructions are performed by directly applying either filtered back projection (FBP) or the simultaneous iterative reconstruction technique (SIRT) to the data. Since HAADF-STEM tomography is a limited angle tomography modality with low signal to noise ratio, these methods can result in significant artifacts in the reconstructed volume. In this paper, we develop a model based iterative reconstruction algorithm for HAADF-STEM tomography. We combine a model for image formation in HAADF-STEM tomography along with a prior model to formulate the tomographic reconstruction as a maximum a posteriori probability (MAP) estimation problem. Our formulation also accounts for certain missing measurements by treating them as nuisance parameters in the MAP estimation framework. We adapt the iterative coordinate descent algorithm to develop an efficient method to minimize the corresponding MAP cost function. Reconstructions of simulated as well as experimental data sets show results that are superior to FBP and SIRT reconstructions, significantly suppressing artifacts and enhancing contrast.

Segmentation propagation, similar to tracking, is the problem of transferring a segmentation of an image to a neighboring image in a sequence. This problem is of particular importance to materials science, where the accurate segmentation of a series of 2D serial-sectioned images of multiple, contiguous 3D structures has important applications. Such structures may have distinct shape, appearance, and topology, which can be considered to improve segmentation accuracy. For example, some materials images may have structures with a specific shape or appearance in each serial section slice, which only changes minimally from slice to slice, and some materials may exhibit specific inter-structure topology that constrains their neighboring relations. Some of these properties have been individually incorporated to segment specific materials images in prior work. In this paper, we develop a propagation framework for materials image segmentation where each propagation is formulated as an optimal labeling problem that can be efficiently solved using the graph-cut algorithm. Our framework makes three key contributions: 1) a homomorphic propagation approach, which considers the consistency of region adjacency in the propagation; 2) incorporation of shape and appearance consistency in the propagation; and 3) a local non-homomorphism strategy to handle newly appearing and disappearing substructures during this propagation. To show the effectiveness of our framework, we conduct experiments on various 3D materials images, and compare the performance against several existing image segmentation methods.

We investigate the automated segmentation of microstructures of a nickel-based superalloy using digital microscopy data. We study the combination of a region merging segmentation method called the stabilized inverse diffusion equation (SIDE), and a stochastic segmentation method, the expectation-maximization/maximization of the posterior marginals (EM/MPM) algorithm. We use the SIDE algorithm to segment the grain boundaries and we use the EM/MPM algorithm to classify two phases of the material within each grain. Experimental results demonstrate the effectiveness of our approach.