Data compression is becoming critical for storing scientific data because many scientific applications need to store large amounts of data and post process this data for scientific discovery. Unlike image and video compression algorithms that limit errors to primary data (PD), scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). This paper presents a physics-informed compression technique implemented as an end-to-end, scalable, GPU-based pipeline for data compression that addresses this requirement. Our hybrid compression technique combines machine learning techniques and standard compression methods. Specifically, we combine an autoencoder, an error-bounded lossy compressor to provide guarantees on raw data error, and a constraint satisfaction post-processing step to preserve the QoIs within a minimal error (generally less than floating point error). The effectiveness of the data compression pipeline is demonstrated by compressing nuclear fusion simulation data generated by a large-scale fusion code, XGC, which produces hundreds of terabytes of data in a single day. Our approach works within the ADIOS framework and results in compression by a factor of more than 150 while requiring only a few percent of the computational resources necessary for generating the data, making the overall approach highly effective for practical scenarios.

Checkpoint/Restart (C/R) strategies are vital for fault tolerance in PDE-based scientific simulations, yet traditional checkpointing incurs significant I/O overhead. Lossy compression offers a scalable solution by reducing checkpoint data size, but conventional methods often lack control over physical invariants (e.g., energy), leading to instability such as oscillations or divergence in Partial Differential Equations (PDE) systems. This paper introduces a stability-preserving compression approach tailored for PDE simulations by explicitly controlling kinetic and potential energy perturbations to ensure stable restarts. Extensive experiments conducted across diverse PDE configurations demonstrate that our method maintains numerical stability with minimal error magnification—even across multiple checkpoint-restart cycles—outperforming state-of-the-art lossy compressors. Parallel evaluations on the Frontier supercomputer show up to 8.4× improvement in checkpoint write performance and 6.3× in read performance, while maintaining relative L2 errors ∼ 2e-6 throughout continued simulation. These results provide practical guidance for balancing compression accuracy, stability, and computational efficiency in large-scale PDE applications.

Generative models have demonstrated strong performance in conditional settings and can be viewed as a form of data compression, where the condition serves as a compact representation. However, their limited controllability and reconstruction accuracy restrict their practical application to data compression. In this work, we propose an efficient latent diffusion framework that bridges this gap by combining a variational autoencoder with a conditional diffusion model. Our method compresses only a small number of keyframes into latent space and uses them as conditioning inputs to reconstruct the remaining frames via generative interpolation, eliminating the need to store latent representations for every frame. This approach enables accurate spatiotemporal reconstruction while significantly reducing storage costs. Experimental results across multiple datasets show that our method achieves up to 10× higher compression ratios than rule-based state-of-the-art compressors such as SZ3, and up to 63% better performance than leading learning-based methods under the same reconstruction error.

Traffic congestion poses significant economic, environmental, and social challenges. High-resolution loop detector data and signal state records from Automated Traffic Signal Performance Measures (ATSPM) offer new opportunities for traffic signal optimization at intersections. However, additional factors such as geometry, traffic volumes, Turning-Movement Counts (TMCs), and human driving behaviors complicate this task. Existing simulators (e.g., SUMO, Vissim) are computationally intensive, while machine learning models often lack lane-specific traffic flow estimation. To address these issues, we propose two computationally efficient Attentional Graph Auto-Encoder frameworks as “Digital Twins” for urban traffic intersections. Leveraging graph representations and Graph Attention Networks (GAT), our models capture lane-level traffic flow dynamics at entry and exit points while remaining agnostic to intersection topology and lane configurations. Trained on over 40,000 hours of realistic traffic simulations with affordable GPU parallelization, our framework produces fine-grained traffic flow time series. This output supports critical applications such as estimating Measures of Effectiveness (MOEs), scaling to urban freeway corridors, and integrating with signal optimization frameworks for improved traffic management.

Scientific applications in fields such as high energy physics, computational fluid dynamics, and climate science generate vast amounts of data at high velocities. This exponential growth in data production is surpassing the advancements in computing power, network capabilities, and storage capacities. To address this challenge, data compression or reduction techniques are crucial. These scientific datasets have underlying data structures that consist of structured and block structured multidimensional meshes where each grid point corresponds to a tensor. It is important that data reduction techniques leverage strong spatial and temporal correlations that are ubiquitous in these applications. Additionally, applications such as CFD, process tensors comprising hundred plus species and their attributes at each grid point. Reduction techniques should be able to leverage interrelationships between the elements in each tensor.In this paper, we propose an attention-based hierarchical compression method utilizing a block-wise compression setup. We introduce an attention-based hyper-block autoencoder to capture inter-block correlations, followed by a block-wise encoder to capture block-specific information. A PCA-based post-processing step is employed to guarantee error bounds for each data block. Our method effectively captures both spatiotemporal and inter-variable correlations within and between data blocks. Compared to the state-of-the-art SZ3, our method achieves up to 8× higher compression ratio on the multi-variable S3D dataset. When evaluated on single-variable setups using the E3SM and XGC datasets, our method still achieves up to 3× and 2× higher compression ratio, respectively.

Scientists conduct large-scale simulations to compute derived quantities from primary data. Thus, it is crucial that data compression techniques maintain bounded errors on these derived quantities or quantities of interest (QoI). Despite the notable performance of recent learned image/video compression approaches using neural networks, they do not guarantee reconstruction errors and cannot manage QoI. This work introduces the Guaranteed Autoencoder with Preserved QoI (GAEQ), which utilizes the interpretation that neural networks with piecewise linear units (PLUs) can be interpreted as a set of linear operators [1] . Although the operators are instance-specific, many instances share the same operator if they fall into the same region of the tessellation formed by PLUs.

Scientists conduct large-scale simulations to compute derived quantities from primary data. Thus, it is crucial that data compression techniques maintain bounded errors on these derived quantities or quantities of interest (QOI). For many spatiotemporal applications, these QOIs are binary in nature and represent presence or absence of a physical phenomenon. In this work, we propose to use a hybrid approah for differential compression for such applications. We use a neural network (NN) approach to determine regions-of-interest (ROIs) where the binary QOIs are going to be prevalent. This is then used with traditional approaches that compress at a lower level (and higher accuracy) for these ROIs as compared to other regions.

Data compression plays a key role in reducing storage and I/O costs. Traditional lossy methods primarily target data on rectilinear grids and cannot leverage the spatial coherence in unstructured mesh data, leading to suboptimal compression ratios. We present a multi-component, error-bounded compression framework designed to enhance the compression of floating-point unstructured mesh data, which is common in scientific applications. Our approach involves interpolating mesh data onto a rectilinear grid and then separately compressing the grid interpolation and the interpolation residuals. This method is general, independent of mesh types and typologies, and can be seamlessly integrated with existing lossy compressors for improved performance. We evaluated our framework across twelve variables from two synthetic datasets and two real-world simulation datasets. The results indicate that the multi-component framework consistently outperforms state-of-the-art lossy compressors on unstructured data, achieving, on average, a 2.3 − 3.5× improvement in compression ratios, with error bounds ranging from 1 × 10 the−6to 1×10−2. We further investigate impact of hyperparameters, such as grid spacing and error allocation, to deliver optimal compression ratios in diverse datasets.

Intersections are conflict zones where the paths of vehicles and pedestrians intersect. They are particularly prone to accidents, with a significant portion of both fatal and nonfatal crashes occurring at these locations. Understanding the diverse behaviors of traffic participants at intersections is crucial for mitigating risks and improving safety. However, conducting experiments in real-world settings to study such behaviors is impractical and hazardous. In this paper, we propose a data-driven approach for predicting and simulating vehicular and pedestrian traffic at signalized intersections. Our method focuses on accurately modeling their behavior and interactions at a busy intersection. We introduce a multi-agent trajectory prediction model, based on the Conditional Variational Autoencoder (CVAE) framework, to generate plausible future trajectories. By incorporating intersection geometry, agent history, and signal state, our model produces a realistic prediction of traffic dynamics. Our model outperforms a strong baseline model—Trajectron++ [1]—by 17% in terms of final displacement errors. We then apply a rule-based trajectory sampling approach to simulate long-horizon behaviors, enabling the proactive identification of potential risks and the exploration of counterfactual scenarios. This research contributes to the development of predictive traffic analytics systems, facilitating safer and more efficient intersection management strategies.

Unconditional scene inference and generation are challenging to learn jointly with a single compositional model. Despite encouraging progress on models that extract object-centric representations ("slots") from images, unconditional generation of scenes from slots has received less attention. This is primarily because learning the multiobject relations necessary to imagine coherent scenes is difficult. We hypothesize that most existing slot-based models have a limited ability to learn object correlations. We propose two improvements that strengthen object correlation learning. The first is to condition the slots on a global, scene-level variable that captures higher-order correlations between slots. Second, we address the fundamental lack of a canonical order for objects in images by proposing to learn a consistent order to use for the autoregressive generation of scene objects. Specifically, we train an autoregressive slot prior to sequentially generate scene objects following a learned order. Ordered slot inference entails first estimating a randomly ordered set of slots using existing approaches for extracting slots from images, then aligning those slots to ordered slots generated autoregressively with the slot prior. Our experiments across three multiobject environments demonstrate clear gains in unconditional scene generation quality. Detailed ablation studies are also provided that validate the two proposed improvements.

Most existing perception systems rely on sensory data acquired from cameras, which perform poorly in low light and adverse weather conditions. To resolve this limitation, we have witnessed advanced LiDAR sensors become popular in perception tasks in autonomous driving applications. Nevertheless, their usage in traffic monitoring systems is less ubiquitous. We identify two significant obstacles in cost-effectively and efficiently developing such a LiDAR-based traffic monitoring system: (i) public LiDAR datasets are insufficient for supporting perception tasks in infrastructure systems, and (ii) 3D annotations on LiDAR point clouds are time-consuming and expensive. To fill this gap, we present an efficient semi-automated annotation tool that automatically annotates LiDAR sequences with tracking algorithms while offering a fully annotated infrastructure LiDAR dataset—FLORIDA (Florida LiDAR-based Object Recognition and Intelligent Data Annotation)—which will be made publicly available. Our advanced annotation tool seamlessly integrates multi-object tracking (MOT), single-object tracking (SOT), and suitable trajectory post-processing techniques. Specifically, we introduce a human-in-the-loop schema in which annotators recursively fix and refine annotations imperfectly predicted by our tool and incrementally add them to the training dataset to obtain better SOT and MOT models. By repeating the process, we significantly increase the overall annotation speed by$3- 4$times and obtain better qualitative annotations than a state-of-the-art annotation tool. The human annotation experiments verify the effectiveness of our annotation tool. In addition, we provide detailed statistics and object detection evaluation results for our dataset in serving as a benchmark for perception tasks at traffic intersections.

In scientific data compression, it is crucial to preserve Quantities of Interest (QoI) derived from the data for accurate post-analysis of scientific applications. In this work, we present Constrained Autoencoders (CAEs) where we impose linear QoI as constraints on neural network activations. We circumvent the difficulty of using standard convex optimization methods on the output predictor in the context of autoencoder-driven compression.

Data compression is becoming critical for data-intensive scientific applications. Scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). Prior work has shown that a pipeline can be built to guarantee error on the primary data (PD) within user-defined bounds and achieve near-floating point QoI errors. In this paper, we present novel computational approaches for accelerating the pipeline and demonstrate results that enable concurrent execution of compression in parallel with the simulation nodes. This allows compression, including the writing of the required compression data, for the previous time step to be completed while the simulation proceeds with the current time step. Overall, the approach presented in this paper results in a 6–8 times improvement in computational overhead compared to previous work. These results were obtained using data generated by a large-scale fusion code called XGC, which produces hundreds of terabytes of data in a single day.

Scientific discoveries are increasingly constrained by limited storage space and I/O capacities. For time-series simulations and experiments, their data often need to be decimated over timesteps to accommodate storage and I/O limitations. In this paper, we propose a technique that addresses storage costs while improving post-analysis accuracy through spatiotemporal adaptive, error-controlled lossy compression. We investigate the trade-off between data precision and temporal output rates, revealing that reducing data precision and increasing timestep frequency lead to more accurate analysis outcomes. Additionally, we integrate spatiotemporal feature detection with data compression and demonstrate that performing adaptive error-bounded compression in higher dimensional space enables greater compression ratios, leveraging the error propagation theory of a transformation-based compressor. To evaluate our approach, we conduct experiments using the well-known E3SM climate simulation code and apply our method to compress variables used for cyclone tracking. Our results show a significant reduction in storage size while enhancing the quality of cyclone tracking analysis, both quantitatively and qualitatively, in comparison to the prevalent timestep decimation approach. Compared to three state-of-the-art lossy compressors lacking feature preservation capabilities, our adaptive compression framework improves perfectly matched cases in TC tracking by 26.4-51.3% at medium compression ratios and by 77.3-571.1% at large compression ratios, with a merely 5–11% computational overhead.

Many scientific simulations and experiments generate terabytes to petabytes of data daily, necessitating data compression techniques. Unlike video and image compression, scientists require methods that accurately preserve primary data (PD) and derived quantities of interest (QoIs). In our previous work, we demonstrated the effectiveness of hybrid compression techniques that combine machine learning with traditional approaches. This paper presents innovative computational techniques aimed at expediting the compression pipeline. Our experiments, conducted on two distinct platforms with a large-scale XGC-based fusion simulation, demonstrate that the overhead incurred by these new approaches is less than one percent of the computational resources needed for the simulation.

We present a new approach to unsupervised shape correspondence learning between pairs of point clouds. We make the first attempt to adapt the classical locally linear embedding algorithm (LLE)-originally designed for nonlinear dimensionality reduction-for shape correspondence. The key idea is to find dense correspondences between shapes by first obtaining high-dimensional neighborhood-preserving embeddings of low-dimensional point clouds and subsequently aligning the source and target embeddings using locally linear transformations. We demonstrate that learning the embedding using a new LLE-inspired point cloud reconstruction objective results in accurate shape correspondences. More specifically, the approach comprises an end-to-end learnable framework of extracting high-dimensional neighborhood-preserving embeddings, estimating locally linear transformations in the embedding space, and reconstructing shapes via divergence measure-based alignment of probability density functions built over reconstructed and target shapes. Our approach enforces embeddings of shapes in correspondence to lie in the same universal/canonical embedding space, which eventually helps regularize the learning process and leads to a simple nearest neighbors approach between shape embeddings for finding reliable correspondences. Comprehensive experiments show that the new method makes noticeable improvements over state-of-the-art approaches on standard shape correspondence benchmark datasets covering both human and nonhuman shapes.

Estimating Arterial Travel Time distributions from high-resolution loop detector data (signal events and vehicle detection events) is a challenging task. Even though high-resolution loop detector data for several years may be available, the lack of other data modes and ground-truth labels, hinders approaches that rely on additional information. Among the approaches relying exclusively on loop detector data, are deterministic physics-based approaches (which use the mechanics of a virtual probe moving down a signalized arterial) and cost-minimization approaches (which estimate travel times from detector counts directly). In this work, we propose and evaluate a hybrid model that uses virtual probe trajectories to indicate probable arrival and departure windows, within which we apply a sequence alignment algorithm on high-resolution loop detector data to match platoons. We generate a broad range of traffic conditions and signal timing plans using SUMO traffic simulator and evaluate our approaches. We also verify our approach using real-world data collected along a 5-intersection signalized corridor. We show that virtual probe trajectories can be replaced by data collected from real probes (when available), thereby improving accuracy. Results show that our approaches enable us to calculate a good estimate of the arterial travel time distribution and are robust to noise. Thus, our methods can be used both with standalone archived loop detector data and in conjunction with data from connected vehicles.

In this paper, we present a new self-supervised scene flow estimation approach for a pair of consecutive point clouds. The key idea of our approach is to represent discrete point clouds as continuous probability density functions using Gaussian mixture models. Scene flow estimation is therefore converted into the problem of recovering motion from the alignment of probability density functions, which we achieve using a closed-form expression of the classic Cauchy-Schwarz divergence. Unlike existing nearest-neighbor-based approaches that use hard pairwise correspondences, our proposed approach establishes soft and implicit point correspondences between point clouds and generates more robust and accurate scene flow in the presence of missing correspondences and outliers. Comprehensive experiments show that our method makes noticeable gains over the Chamfer Distance and the Earth Mover’s Distance in real-world environments and achieves state-of-the-art performance among self-supervised learning methods on FlyingThings3D and KITTI, even outperforming some supervised methods with ground truth annotations.

Efficient data compression is becoming increasingly critical for storing scientific data because many scientific applications produce vast amounts of data. This paper presents an end-to-end algorithmic and software pipeline for data compression that guarantees both error bounds on primary data (PD) and derived data, known as Quantities of Interest (QoI).We demonstrate the effectiveness of the pipeline by compressing fusion data generated by a large-scale fusion code, XGC, which produces tens of petabytes of data in a single day. We demonstrate that the compression is conducted by setting aside computational resources known as staging nodes, and does not impact the simulation performance. For efficient parallel I/O, the pipeline uses ADIOS2, which many codes such as XGC already use for their parallel I/O. We show that our approach can compress the data by two orders of magnitude while guaranteeing high accuracy on both the PD and the QoIs. Further, the amount of resources required by compression is a few percent of the resources required by simulation while ensuring that the compression time for each stage is less than the corresponding simulation time.This pipeline consists of three main steps. The first step decomposes the data using domain decomposition into small subdomains. Each subdomain is then compressed independently to achieve a high level of parallelism. The second step uses existing techniques that guarantee error bounds on the primary data for each subdomain. The third step uses a post-processing optimization technique based on Lagrange multipliers to reduce the QoI errors for data corresponding to each subdomain. The Lagrange multipliers generated can be further quantized or truncated to increase the compression level. All of the above characteristics of our approach make it highly practical to apply on-the-fly compression while guaranteeing errors on QoIs that are critical to the scientists.

The increase of computer processing speed is significantly outpacing improvements in network and storage bandwidth, leading to the big data challenge in modern science, where scientific applications can quickly generate much more data than that can be transferred and stored. As a result, big scientific data must be reduced by a few orders of magnitude while the accuracy of the reduced data needs to be guaranteed for further scientific explorations. Moreover, scientists are often interested in some specific spatial/temporal regions in their data, where higher accuracy is required. The locations of the regions requiring high accuracy can sometimes be prescribed based on application knowledge, while other times they must be estimated based on general spatial/temporal variation. In this paper, we develop a novel multilevel approach which allows users to impose region-wise compression error bounds. Our method utilizes the byproduct of a multilevel compressor to detect regions where details are rich and we provide the theoretical underpinning for region-wise error control. With spatially varying precision preservation, our approach can achieve significantly higher compression ratios than single-error bounded compression approaches and control errors in the regions of interest.

Traffic flow dynamics in the vicinity of urban arterial intersections is a complex and nonlinear phenomenon, influenced by factors such as signal timing plan, road geometry, driver behaviors, etc. Predicting such flow dynamics is an important task for urban traffic signal control and planning. Current methods use microscopic simulation for studying the impact of a large number of signal timing plans at each of the intersections. A major drawback of microscopic simulation is that they are they are based on source destination traffic generation models and cannot incorporate the high resolution loop detector data such as that are provided by automated traffic signal performance measures (ATSPM) based systems. The arrival (or departure) information of each vehicle on a detector can be thought of as a time series waveform. Given the high granularity of ATSPM data, this waveform can be used to several interesting analyses. The waveforms can be used to derive information on platoon dispersion as vehicles progress across the corridor. Also, these waveforms can be modelled to understand how the vehicles progress across the corridor for a variety of signal timing plans. In this paper, we show that deep neural networks based machine learning systems can be used to effectively leverage the waveforms collected at multiple sensors (stopbar and advanced) on the intersection to model the traffic dynamics both at an intersection and across intersections. We show that modelling of these waveforms can be useful to understand traffic flow dynamics under different signal timing plans and can be potentially integrated into signal timing optimization software. Further, these methods are three to four orders of magnitudes faster than using microscopic simulations.

Unsupervised multi-object representation learning depends on inductive biases to guide the discovery of object-centric representations that generalize. However, we observe that methods for learning these representations are either impractical due to long training times and large memory consumption or forego key inductive biases. In this work, we introduce EfficientMORL, an efficient framework for the unsupervised learning of object-centric representations. We show that optimization challenges caused by requiring both symmetry and disentanglement can in fact be addressed by high-cost iterative amortized inference by designing the framework to minimize its dependence on it. We take a two-stage approach to inference: first, a hierarchical variational autoencoder extracts symmetric and disentangled representations through bottom-up inference, and second, a lightweight network refines the representations with top-down feedback. The number of refinement steps taken during training is reduced following a curriculum, so that at test time with zero steps the model achieves 99.1% of the refined decomposition performance. We demonstrate strong object decomposition and disentanglement on the standard multi-object benchmark while achieving nearly an order of magnitude faster training and test time inference over the previous state-of-the-art model.

In this paper, we present a novel and effective approach to truck and trailer classification, which integrates deep learning models and conventional image processing and computer vision techniques. The developed method groups trucks into subcategories by carefully examining the truck classes and identifying key geometric features for discriminating truck and trailer types. We also present three discriminating features that involve shape, texture, and semantic information to identify trailer types. Experimental results demonstrate that the developed hybrid approach can achieve high accuracy with limited training data, where the vanilla deep learning approaches show moderate performance due to over-fitting and poor generalization. Additionally, the models generated are human-understandable.

We propose SparsePipe, an efficient and asynchronous parallelism approach for handling 3D point clouds with multi-GPU training. SparsePipe is built to support 3D sparse data such as point clouds. It achieves this by adopting generalized convolutions with sparse tensor representation to build expressive high-dimensional convolutional neural networks. Compared to dense solutions, the new models can efficiently process irregular point clouds without densely sliding over the entire space, significantly reducing the memory requirements and allowing higher resolutions of the underlying 3D volumes for better performance. SparsePipe exploits intra-batch parallelism that partitions input data into multiple processors and further improves the training throughput with inter-batch pipelining to overlap communication and computing. Besides, it suitably partitions the model when the GPUs are heterogeneous such that the computing is load-balanced with reduced communication overhead. Using experimental results on an eight-GPU platform, we show that SparsePipe can parallelize effectively and obtain better performance on current point cloud benchmarks for both training and inference, compared to its dense solutions.

Combining a hyperspectral (HS) image and a multispectral (MS) image - an example of image fusion - can result in a spatially and spectrally high-resolution image. Despite the plethora of fusion algorithms in remote sensing, a necessary prerequisite, namely registration, is mostly ignored. This limits their application to well-registered images from the same source. In this article, we propose and validate an integrated registration and fusion approach (code available at https://github.com/zhouyuanzxcv/Hyperspectral). The registration algorithm minimizes a least-squares (LSQ) objective function with the point spread function (PSF) incorporated together with a nonrigid freeform transformation applied to the HS image and a rigid transformation applied to the MS image. It can handle images with significant scale differences and spatial distortion. The fusion algorithm takes the full high-resolution HS image as an unknown in the objective function. Assuming that the pixels lie on a low-dimensional manifold invariant to local linear transformations from spectral degradation, the fusion optimization problem leads to a closed-form solution. The method was validated on the Pavia University, Salton Sea, and the Mississippi Gulfport datasets. When the proposed registration algorithm is compared to its rigid variant and two mutual information-based methods, it has the best accuracy for both the nonrigid simulated dataset and the real dataset, with an average error less than 0.15 pixels for nonrigid distortion of maximum 1 HS pixel. When the fusion algorithm is compared with current state-of-the-art algorithms, it has the best performance on images with registration errors as well as on simulations that do not consider registration effects.

In this work, we present a novel, theoretical approach to address one of the longstanding problems in computer vision: 2D and 3D affine invariant feature matching. Our proposed Grassmannian Graph (GrassGraph) framework employs a two stage procedure that is capable of robustly recovering correspondences between two unorganized, affinely related feature (point) sets. In the ideal case, the first stage maps the feature sets to an affine invariant Grassmannian representation, where the features are mapped into the same subspace. It turns out that coordinate representations extracted from the Grassmannian differ by an arbitrary orthonormal matrix. In the second stage, by approximating the Laplace-Beltrami operator (LBO) on these coordinates, this extra orthonormal factor is nullified, providing true affine invariant coordinates which we then utilize to recover correspondences via simple mutual nearest neighbor relations. Our validation benchmarks use large number of experimental trials performed on 2D and 3D datasets. Experimental results show that the proposed GrassGraph method successfully recovers large affine transformations.

In this paper, we present a novel terrain classification framework for large-scale remote sensing images. A well-performing multi-scale superpixel tessellation based segmentation approach is employed to generate homogeneous and irregularly shaped regions, and a transfer learning technique is sequentially deployed to derive representative deep features by utilizing successful pre-trained convolutional neural network (CNN) models. This design is aimed to overcome the big problem of lacking available ground-truth data and to increase the generalization power of the multi-pixel descriptor. In the subsequent classification step, we train a fast and robust support vector machine (SVM) to assign the pixel-level labels. Its maximum-margin property can be easily combined with a graph Laplacian propagation approach. Moreover, we analyze the advantages of applying a feature selection technique to the deep CNN features which are extracted by transfer learning. In the experiments, we evaluate the whole framework based on different geographical types. Compared with other region-based classification methods, the results show that our framework can obtain state-of-the-art performance w.r.t. both classification accuracy and computational efficiency.

A novel algorithm is developed to downscale soil moisture (SM), obtained at satellite scales of 10-40 km to 1 km by utilizing its temporal correlations to historical auxiliary data at finer scales. Including such correlations drastically minimizes the size of the training set needed, accounts for time-lagged relationships, and enables downscaling even in the presence of short gaps in the auxiliary data. The algorithm is based upon bagged regression trees (BRT) and uses correlations between high-resolution remote sensing products and SM observations. The algorithm trains multiple RTs and automatically chooses the trees that generate the best downscaled estimates. The algorithm was evaluated using a multiscale synthetic data set in north central Florida for two years, including two growing seasons of corn and one growing season of cotton per year. The time-averaged error across the region was found to be 0.01 m3/m3, with a standard deviation of 0.012 m3/m3when 0.02% of the data were used for training in addition to temporal correlations from the past seven days, and all available data from the past year. The maximum spatially averaged errors obtained using this algorithm in downscaled SM were 0.005 m3/m3, for pixels with cotton land cover. When land surface temperature (LST) on the day of downscaling was not included in the algorithm to simulate “data gaps,” the spatially averaged error increased minimally by 0.015 m3/m3when LST is unavailable on the day of downscaling. The results indicate that the BRT-based algorithm provides high accuracy for downscaling SM using complex nonlinear spatiotemporal correlations, under heterogeneous micrometeorological conditions.

We introduce a framework in which a stack of images is considered to be a 2-D parametric surface embedded in a higher dimensional space. This is a simple yet powerful idea, known in the literature but not exploited to its fullest. We discuss the properties of image stacks as parametric surfaces, apply this framework to image registration by presenting the image stack surface relative area (ISSRA) registration measure. We show the power of ISSRA as an effective objective function for image registration. Essentially, it shows good performance across a variety of different categories of registration problems: pairwise, groupwise, affine, and non-rigid. Mutual information (MI)-a classical and effective approach for registration-is widely considered to be a good choice for multimodal and pairwise registration while being difficult to extend to the groupwise setting. We discuss the deficiency of MI in the groupwise case from a theoretical point of view, present its connection to ISSRA in the pairwise case, and then show the ready extensibility of ISSRA to the groupwise setting. Experiments and comparisons are performed on different categories of image registration to showcase ISSRA's wide range of applicability to registration problems in practice.

Hyperspectral unmixing while considering endmember variability is usually performed by the normal compositional model, where the endmembers for each pixel are assumed to be sampled from unimodal Gaussian distributions. However, in real applications, the distribution of a material is often not Gaussian. In this paper, we use Gaussian mixture models (GMM) to represent endmember variability. We show, given the GMM starting premise, that the distribution of the mixed pixel (under the linear mixing model) is also a GMM (and this is shown from two perspectives). The first perspective originates from random variable transformations and gives a conditional density function of the pixels given the abundances and GMM parameters. With proper smoothness and sparsity prior constraints on the abundances, the conditional density function leads to a standard maximum a posteriori (MAP ) problem which can be solved using generalized expectation maximization. The second perspective originates from marginalizing over the endmembers in the GMM, which provides us with a foundation to solve for the endmembers at each pixel. Hence, compared to the other distribution based methods, our model can not only estimate the abundances and distribution parameters, but also the distinct endmember set for each pixel. We tested the proposed GMM on several synthetic and real datasets, and showed its potential by comparing it to current popular methods.

In this study, a data-fusion algorithm is developed for estimation of high-resolution brightness temperatures (TB) at 1km from Soil Moisture Active Passive (SMAP) fine-grid TBproduct at 9km. It uses image segmentation to spatio-temporally cluster the study region based on meteorological and land cover similarity, followed by a support vector machine based regression that computes the value of the high-resolution TBat all pixels. High resolution remote sensing products such as land surface temperature, normalized difference vegetation index, enhanced vegetation index, precipitation, soil texture, and land-cover were used for disaggregation. The algorithm was implemented in Iowa, United States, from May to September 2016, and compared with the field observations of TBfrom Microwave Water and Energy Balance Experiment conducted as a part of the Soil Moisture Active Passive Validation Experiment (SMAPVEX16-MicroWEX). Additionally, they were also compared with the Sentinel downscaled SMAP TBat 1km. High resolution soil moisture is subsequently derived from high resolution TBusing inverse models.

Considering the graph of a feature function as an embedded surface in three dimensions is a standard device in computer vision. When multiple feature functions (eg. multiple images) are available, the natural extension of the above concept is to a higher-dimensional embedded surface. This has received surprisingly little attention. In this paper, we advocate for this view by showing the utility of surface area for estimating spatial transformations between images for the purposes of registration. In contrast to the entropy of the stack of images (when represented as continuous and differentiable functions) being zero, the surface area turns out to be useful measure for image registration. We show that during the progression of standard registration algorithms like Congealing, the area of the stack of images being registered decreases. Our own algorithm for affine registration based on image stack surface area (ISSA) minimization has several advantages compared with Congealing and Mutual Information (MI) registration. Finally, we highlight the generality of our framework by also showcasing experiments on affine point-set registration.

In this paper, a novel machine learning algorithm is presented for disaggregation of satellite soil moisture (SM) based on self-regularized regressive models (SRRMs) using high-resolution correlated information from auxiliary sources. It includes regularized clustering that assigns soft memberships to each pixel at a fine scale followed by a kernel regression that computes the value of the desired variable at all pixels. Coarse-scale remotely sensed SM was disaggregated from 10 to 1 km using land cover (LC), precipitation, land surface temperature, leaf area index, and in situ observations of SM. This algorithm was evaluated using multiscale synthetic observations in NC Florida for heterogeneous agricultural LCs. It was found that the rmse for 96% of the pixels was less than 0.02 m3/m3. The clusters generated represented the data well and reduced the rmse by up to 40% during periods of high heterogeneity in LC and meteorological conditions. The Kullback-Leibler divergence (KLD) between the true SM and the disaggregated estimates is close to zero, for both vegetated and bare-soil LCs. The disaggregated estimates were compared with those generated by the principle of relevant information (PRI) method. The rmse for the PRI disaggregated estimates is higher than the rmse for the SRRM on each day of the season. The KLD of the disaggregated estimates generated by the SRRM is at least four orders of magnitude lower than those for the PRI disaggregated estimates, whereas the computational time needed was reduced by three times. The results indicate that the SRRM can be used for disaggregating SM with complex nonlinear correlations on a grid with high accuracy.

The normal compositional model (NCM) has been extensively used in hyperspectral unmixing. However, previous research has mostly focused on estimation of endmembers and/or their variability, based on the assumption that the pixels are independent random variables. In this paper, we show that this assumption does not hold if all the pixels are generated by a fixed endmember set. This introduces another concept, endmember uncertainty, which is related to whether the pixels fit into the endmember simplex. To further develop this idea, we derive the NCM from the ground up without the pixel independence assumption, along with (i) using different noise levels at different wavelengths and (ii) using a spatial and sparsity promoting prior for the abundances. The resulting new formulation is called the spatial compositional model (SCM) to better differentiate it from the NCM. The SCM maximum a posteriori (MAP) objective leads to an optimization problem featuring noise weighted least-squares minimization for unmixing. The problem is solved by projected gradient descent, resulting in an algorithm that estimates endmembers, abundances, noise variances, and endmember uncertainty simultaneously. We compared SCM with current state-of-the-art algorithms on synthetic and real images. The results show that SCM can in the main provide more accurate endmembers and abundances. Moreover, the estimated uncertainty can serve as a prediction of endmember error under certain conditions.

Although vortex analysis and detection have been extensively investigated in the past, none of the existing techniques are able to provide fully robust and reliable identification results. Local vortex detection methods are popular as they are efficient and easy to implement, and produce binary outputs based on a user-specified, hard threshold. However, vortices are global features, which present challenges for local detectors. On the other hand, global detectors are computationally intensive and require considerable user input. In this work, we propose a consensus-based uncertainty model and introduce spatial proximity to enhance vortex detection results obtained using point-based methods. We use four existing local vortex detectors and convert their outputs into fuzzy possibility values using a sigmoid-based soft-thresholding approach. We apply a majority voting scheme that enables us to identify candidate vortex regions with a higher degree of confidence. Then, we introduce spatial proximity- based analysis to discern the final vortical regions. Thus, by using spatial proximity coupled with fuzzy inputs, we propose a novel uncertainty analysis approach for vortex detection. We use expert's input to better estimate the system parameters and results from two real-world data sets demonstrate the efficacy of our method.

An novel algorithm is proposed to downscale microwave brightness temperatures (TB), at scales of 10-40 km such as those from the Soil Moisture Active Passive mission to a resolution meaningful for hydrological and agricultural applications. This algorithm, called Self-Regularized Regressive Models (SRRM), uses auxiliary variables correlated to TBalong-with a limited set of in-situ SM observations, which are converted to high resolution TBobservations using biophysical models. It includes an information-theoretic clustering step based on all auxiliary variables to identify areas of similarity, followed by a kernel regression step that produces downscaled TB. This was implemented on a multi-scale synthetic data-set over NC-Florida for one year. An RMSE of 5.76 K with standard deviation of 2.8 K was achieved during the vegetated season and an RMSE of 1.2 K with a standard deviation of 0.9 K during periods of no vegetation.

Urban neighborhood classification using very high resolution (VHR) remote sensing imagery is a challenging and {\em emerging} application. A semi-supervised learning approach for identifying neighborhoods is presented which employs superpixel tessellation representations of VHR imagery. The image representation utilizes homogeneous and irregularly shaped regions termed superpixels and derives novel features based on intensity histograms, geometry, corner and superpixel density and scale of tessellation. The semi-supervised learning approach uses a support vector machine (SVM) to obtain a preliminary classification which is then subsequently refined using graph Laplacian propagation. Several intermediate stages in the pipeline are presented to showcase the important features of this approach. We evaluated this approach on four different geographic settings with varying neighborhood types and compared it with the recent Gaussian Multiple Learning algorithm. This evaluation shows several advantages, including model building, accuracy, and efficiency which makes it a great choice for deployment in large scale applications like global human settlement mapping and population distribution (e.g., LandScan), and change detection.

In this paper, we cast the problem of point cloud matching as a shape matching problem by transforming each of the given point clouds into a shape representation called the Schrödinger distance transform (SDT) representation. This is achieved by solving a static Schrödinger equation instead of the corresponding static Hamilton-Jacobi equation in this setting. The SDT representation is an analytic expression and following the theoretical physics literature, can be normalized to have unit 2 norm - making it a square-root density, which is identified with a point on a unit Hilbert sphere, whose intrinsic geometry is fully known. The Fisher-Rao metric, a natural metric for the space of densities leads to analytic expressions for the geodesic distance between points on this sphere. In this paper, we use the well known Riemannian framework never before used for point cloud matching, and present a novel matching algorithm. We pose point set matching under rigid and non-rigid transformations in this framework and solve for the transformations using standard nonlinear optimization techniques. Finally, to evaluate the performance of our algorithm - dubbed SDTM - we present several synthetic and real data examples along with extensive comparisons to state-of-the-art techniques. The experiments show that our algorithm outperforms state-of the-art point set registration algorithms on many quantitative metrics.

The past twenty years has seen the explosion of the "shape zoo": myriad shape representations, each with pros and cons. Of the varied denizens, distance transforms and density function shape representations have proven to be the most utile. Distance transforms inherit the numerous geometric advantages of implicit curve representations while density functions are unmatched in their approach to the modeling of uncertainty and noise in shape features. We have not seen much rapprochement between these two representations in general. In this work, we introduce a complex wave representation (CWR) of shape which has the ability to simultaneously carry probabilistic information via its magnitude and geometric information via its phase, achieving an integration of distance transforms and density function shape representations. The CWR is a parametric representation with cluster centers akin to a mixture model and curve normal information akin to signed distance functions. We demonstrate the perceptual gains of the CWR, highlight the advantages of the probabilistic aspect for noisy shape alignment by a likelihood approach, and fusing both aspects we show that the CWR leads to a feature space in which kernel PCA yields approximate closed curves and probability density functions.

Abstract Robust automated vortex detection algorithms are needed to facilitate the exploration of large‐scale turbulent fluid flow simulations. Unfortunately, robust non‐local vortex detection algorithms are computationally intractable for large data sets and local algorithms, while computationally tractable, lack robustness. We argue that the deficiencies inherent to the local definitions occur because of two fundamental issues: the lack of a rigorous definition of a vortex and the fact that a vortex is an intrinsically non‐local phenomenon. As a first step towards addressing this problem, we demonstrate the use of machine learning techniques to enhance the robustness of local vortex detection algorithms. We motivate the presence of an expert‐in‐the‐loop using empirical results based on machine learning techniques. We employ adaptive boosting to combine a suite of widely used, local vortex detection algorithms, which we term weak classifiers, into a robust compound classifier. Fundamentally, the training phase of the algorithm, in which an expert manually labels small, spatially contiguous regions of the data, incorporates non‐local information into the resulting compound classifier. We demonstrate the efficacy of our approach by applying the compound classifier to two data sets obtained from computational fluid dynamical simulations. Our results demonstrate that the compound classifier has a reduced misclassification rate relative to the component classifiers.

In this paper, we propose a very simple and elegant patch-based, machine learning technique for image denoising using the higher order singular value decomposition (HOSVD). The technique simply groups together similar patches from a noisy image (with similarity defined by a statistically motivated criterion) into a 3D stack, computes the HOSVD coefficients of this stack, manipulates these coefficients by hard thresholding, and inverts the HOSVD transform to produce the final filtered image. Our technique chooses all required parameters in a principled way, relating them to the noise model. We also discuss our motivation for adopting the HOSVD as an appropriate transform for image denoising. We experimentally demonstrate the excellent performance of the technique on grayscale as well as color images. On color images, our method produces state-of-the-art results, outperforming other color image denoising algorithms at moderately high noise levels. A criterion for optimal patch-size selection and noise variance estimation from the residual images (after denoising) is also presented.

Despite the ubiquitous use of distance transforms in the shape analysis literature and the popularity of fast marching and fast sweeping methods - essentially Hamilton-Jacobi solvers, there is very little recent work leveraging the Hamilton-Jacobi to Schrödinger connection for representational and computational purposes. In this work, we exploit the linearity of the Schrödinger equation to (i) design fast discrete convolution methods using the FFT to compute the distance transform, (ii) derive the histogram of oriented gradients (HOG) via the squared magnitude of the Fourier transform of the wave function, (iii) extend the Schrödinger formalism to cover the case of curves parametrized as line segments as opposed to point-sets, (iv) demonstrate that the Schrödinger formalism permits the addition of wave functions - an operation that is not allowed for distance transforms, and finally (v) construct a fundamentally new Schrödinger equation and show that it can represent both the distance transform and its gradient density - not possible in earlier efforts.

This paper presents Data-Driven Tree-structured Bayesian network (DDT), a novel probabilistic graphical model for hierarchical unsupervised image segmentation. The DDT captures long and short-ranged correlations between neighboring regions in each image using a tree-structured prior. Unlike other previous work, DDT first segments an input image into superpixels and learn a tree-structured prior based on the topology of superpixels in different scales. Such a tree structure is referred to as data-driven tree structure. Each superpixel is represented by a variable node taking a discrete value of class/label of the segmentation. The probabilistic relationships among the nodes are represented by edges in the network. The unsupervised image segmentation, hence, can be viewed as an inference problem of the nodes in the tree structure of DDT, which can be carried out efficiently. We evaluate quantitatively our results with respect to the ground-truth segmentation, demonstrating that our proposed framework performs competitively with the state of the art in unsupervised image segmentation and contour detection.

We describe a nonparametric framework for incorporating information from co-registered anatomical images into positron emission tomographic (PET) image reconstruction through priors based on information theoretic similarity measures. We compare and evaluate the use of mutual information (MI) and joint entropy (JE) between feature vectors extracted from the anatomical and PET images as priors in PET reconstruction. Scale-space theory provides a framework for the analysis of images at different levels of detail, and we use this approach to define feature vectors that emphasize prominent boundaries in the anatomical and functional images, and attach less importance to detail and noise that is less likely to be correlated in the two images. Through simulations that model the best case scenario of perfect agreement between the anatomical and functional images, and a more realistic situation with a real magnetic resonance image and a PET phantom that has partial volumes and a smooth variation of intensities, we evaluate the performance of MI and JE based priors in comparison to a Gaussian quadratic prior, which does not use any anatomical information. We also apply this method to clinical brain scan data using F(18) Fallypride, a tracer that binds to dopamine receptors and therefore localizes mainly in the striatum. We present an efficient method of computing these priors and their derivatives based on fast Fourier transforms that reduce the complexity of their convolution-like expressions. Our results indicate that while sensitive to initialization and choice of hyperparameters, information theoretic priors can reconstruct images with higher contrast and superior quantitation than quadratic priors.

We present a new method for compact representation of large image datasets. Our method is based on treating small patches from a 2-D image as matrices as opposed to the conventional vectorial representation, and encoding these patches as sparse projections onto a set of exemplar orthonormal bases, which are learned a priori from a training set. The end result is a low-error, highly compact image/patch representation that has significant theoretical merits and compares favorably with existing techniques (including JPEG) on experiments involving the compression of ORL and Yale face databases, as well as a database of miscellaneous natural images. In the context of learning multiple orthonormal bases, we show the easy tunability of our method to efficiently represent patches of different complexities. Furthermore, we show that our method is extensible in a theoretically sound manner to higher-order matrices ("tensors"). We demonstrate applications of this theory to compression of well-known color image datasets such as the GaTech and CMU-PIE face databases and show performance competitive with JPEG. Lastly, we also analyze the effect of image noise on the performance of our compression schemes.

This paper proposes a new affine registration algorithm for matching two point sets in IR2or IR3. The input point sets are represented as probability density functions, using either Gaussian mixture models or discrete density models, and the problem of registering the point sets is treated as aligning the two distributions. Since polynomials transform as symmetric tensors under an affine transformation, the distributions' moments, which are the expected values of polynomials, also transform accordingly. Therefore, instead of solving the harder problem of aligning the two distributions directly, we solve the softer problem of matching the distributions' moments. By formulating a least-squares problem for matching moments of the two distributions up to degree three, the resulting cost function is a polynomial that can be efficiently optimized using techniques originated from algebraic geometry: the global minimum of this polynomial can be determined by solving a system of polynomial equations. The algorithm is robust in the presence of noises and outliers, and we validate the proposed algorithm on a variety of point sets with varying degrees of deformation and noise.

Shape matching plays a prominent role in the comparison of similar structures. We present a unifying framework for shape matching that uses mixture models to couple both the shape representation and deformation. The theoretical foundation is drawn from information geometry wherein information matrices are used to establish intrinsic distances between parametric densities. When a parameterized probability density function is used to represent a landmark-based shape, the modes of deformation are automatically established through the information matrix of the density. We first show that given two shapes parameterized by Gaussian mixture models (GMMs), the well-known Fisher information matrix of the mixture model is also a Riemannian metric (actually, the Fisher-Rao Riemannian metric) and can therefore be used for computing shape geodesics. The Fisher-Rao metric has the advantage of being an intrinsic metric and invariant to reparameterization. The geodesicâcomputed using this metricâestablishes an intrinsic deformation between the shapes, thus unifying both shape representation and deformation. A fundamental drawback of the Fisher-Rao metric is that it is not available in closed form for the GMM. Consequently, shape comparisons are computationally very expensive. To address this, we develop a new Riemannian metric based on generalized \phi-entropy measures. In sharp contrast to the Fisher-Rao metric, the new metric is available in closed form. Geodesic computations using the new metric are considerably more efficient. We validate the performance and discriminative capabilities of these new information geometry-based metrics by pairwise matching of corpus callosum shapes. We also study the deformations of fish shapes that have various topological properties. A comprehensive comparative analysis is also provided using other landmark-based distances, including the Hausdorff distance, the Procrustes metric, landmark-based diffeomorphisms, and the bending energies of the thin-plate (TPS) and Wendland splines.

We present a new, geometric approach for determining the probability density of the intensity values in an image. We drop the notion of an image as a set of discrete pixels, and assume a piecewise-continuous representation. The probability density can then be regarded as being proportional to the area between two nearby isocontours of the image surface. Our paper extends this idea to joint densities of image pairs. We demonstrate the application of our method to affine registration between two or more images using information theoretic measures such as mutual information. We show cases where our method outperforms existing methods such as simple histograms, histograms with partial volume interpolation, Parzen windows, etc. under fine intensity quantization for affine image registration under significant image noise. Furthermore, we demonstrate results on simultaneous registration of multiple images, as well as for pairs of volume datasets, and show some theoretical properties of our density estimator. Our approach requires the selection of only an image interpolant. The method neither requires any kind of kernel functions (as in Parzen windows) which are unrelated to the structure of the image in itself, nor does it rely on any form of sampling for density estimation.

Shape representation and retrieval of stored shape models are becoming increasingly more prominent in fields such as medical imaging, molecular biology and remote sensing. We present a novel framework that directly addresses the necessity for a rich and compressible shape representation, while simultaneously providing an accurate method to index stored shapes. The core idea is to represent point-set shapes as the square root of probability densities expanded in a wavelet basis. We then use this representation to develop a natural similarity metric that respects the geometry of these probability distributions, i.e. under the wavelet expansion, densities are points on a unit hypersphere and the distance between densities is given by the separating arc length. The process uses a linear assignment solver for non-rigid alignment between densities prior to matching; this has the connotation of "sliding" wavelet coefficients akin to the sliding block puzzle L'Âne Rouge. We illustrate the utility of this framework by matching shapes from the MPEG-7 data set and provide comparisons to other similarity measures, such as Euclidean distance shape distributions.

We present a new method for compact representation of large image datasets. Our method is based on treating small patches from an image as matrices as opposed to the conventional vectorial representation, and encoding those patches as sparse projections onto a set of exemplar orthonormal bases, which are learned a priori from a training set. The end result is a low-error, highly compact image/patch representation that has significant theoretical merits and compares favorably with existing techniques on experiments involving the compression of ORL and Yale face databases.

Groupwise registration of a set of shapes represented by unlabeled point sets is a challenging problem since, usually, this involves solving for point correspondence in a nonrigid motion setting. In this paper, we propose a novel and robust algorithm that is capable of simultaneously computing the mean shape, represented by a probability density function, from multiple unlabeled point sets(represented by finite-mixture models), and registering them nonrigidly to this emerging mean shape. This algorithm avoids the correspondence problem by minimizing the Jensen-Shannon (JS) divergence between the point sets represented as finite mixtures of Gaussian densities. We motivate the use of the JS divergence by pointing out its close relationship to hypothesis testing. Essentially,minimizing the JS divergence is asymptotically equivalent to maximizing the likelihood ratio formed from a probability density of the pooled point sets and the product of the probability densities of the individual point sets. We derive the analytic gradient of the cost function, namely, the JS-divergence, in order to efficiently achieve the optimal solution. The cost function is fully symmetric, with no bias toward any of the given shapes to be registered and whose mean is being sought. A by-product of the registration process is a probabilistic atlas, which is defined as the convex combination of the probability densities of the input point sets being aligned. Our algorithm can be especially useful for creating atlases of various shapes present in images and for simultaneously (rigidly or nonrigidly)registering 3D range data sets (in vision and graphics applications), without having to establish any correspondence. We present experimental results on nonrigidly registering 2D and 3D real and synthetic data (point sets).

Density estimation for observational data plays an integral role in a broad spectrum of applications, e.g., statistical data analysis and information-theoretic image registration. Of late, wavelet-based density estimators have gained in popularity due to their ability to approximate a large class of functions, adapting well to difficult situations such as when densities exhibit abrupt changes. The decision to work with wavelet density estimators brings along with it theoretical considerations (e.g., non-negativity, integrability) and empirical issues (e.g., computation of basis coefficients) that must be addressed in order to obtain a bona fide density. In this paper, we present a new method to accurately estimate a non-negative density which directly addresses many of the problems in practical wavelet density estimation. We cast the estimation procedure in a maximum likelihood framework which estimates the square root of the density radicalp, allowing us to obtain the natural non-negative density representation ( radicalp)(2). Analysis of this method will bring to light a remarkable theoretical connection with the Fisher information of the density and, consequently, lead to an efficient constrained optimization procedure to estimate the wavelet coefficients. We illustrate the effectiveness of the algorithm by evaluating its performance on mutual information-based image registration, shape point set alignment, and empirical comparisons to known densities. The present method is also compared to fixed and variable bandwidth kernel density estimators.

We propose a novel affine registration algorithm for matching 2D feature points. Unlike many previously published work on affine point matching, the proposed algorithm does not require any optimization and in the absence of data noise, the algorithm will recover the exact affine transformation and the unknown correspondence. The two-step algorithm first reduces the general affine case to the orthogonal case, and the unknown rotation is computed as the roots of a low-degree polynomial with complex coefficients. The algebraic and geometric ideas behind the proposed method are both clear and transparent, and its implementation is straightforward. We validate the algorithm on a variety of synthetic 2D point sets as well as feature points on images of real-world objects

We present a new, robust and computationally efficient method for estimating the probability density of the intensity values in an image. Our approach makes use of a continuous representation of the image and develops a relation between probability density at a particular intensity value and image gradients along the level sets at that value. Unlike traditional sample-based methods such as histograms, minimum spanning trees (MSTs), Parzen windows or mixture models, our technique expressly accounts for the relative ordering of the intensity values at different image locations and exploits the geometry of the image surface. Moreover, our method avoids the histogram binning problem and requires no critical parameter tuning. We extend the method to compute the joint density between two or more images. We apply our density estimation technique to the task of affine registration of 2D images using mutual information and show good results under high noise.

In this paper, we propose a novel and robust algorithm for the groupwise non-rigid registration of multiple unlabeled point-sets with no bias toward any of the given point-sets. To quantify the divergence between multiple probability distributions each estimated from the given point sets, we develop a novel measure based on their cumulative distribution functions that we dub the CDF-JS divergence. The measure parallels the well known Jensen-Shannon divergence (defined for probability density functions) but is more regular than the JS divergence since its definition is based on CDFs as opposed to density functions. As a consequence, CDF-JS is more immune to noise and statistically more robust than the JS.We derive the analytic gradient of the CDF-JS divergence with respect to the non-rigid registration parameters for use in the numerical optimization of the groupwise registration leading a computationally efficient and accurate algorithm. The CDF-JS is symmetric and has no bias toward any of the given point-sets, since there is NO fixed reference data set. Instead, the groupwise registration takes place between the input data sets and an evolving target dubbed the pooled model. This target evolves to a fully registered pooled data set when the CDF-JS defined over this pooled data is minimized. Our algorithm is especially useful for creating atlases of various shapes (represented as point distribution models) as well as for simultaneously registering 3D range data sets without establishing any correspondence. We present experimental results on non-rigid registration of 2D/3D real point set data.

We propose an algorithm achieving the matching of two point sets with unknown correspondence via the joint clustering of the two sets and estimation of a diffeomorphism linking the cluster centers via geodesic splines.

One of the key challenges in deformable shape modeling is the problem of estimating a meaningful average or mean shape from a set of unlabeled shapes. We present a new joint clustering and matching algorithm that is capable of computing such a mean shape from multiple shape samples which are represented by unlabeled point-sets. An iterative bootstrap process is used wherein multiple shape sample point-sets are nonrigidly deformed to the emerging mean shape, with subsequent estimation of the mean shape based on these nonrigid alignments. The process is entirely symmetric with no bias toward any of the original shape sample point-sets. We believe that this method can be especially useful for creating atlases of various shapes present in medical images. We have applied the method to create mean shapes from nine hand-segmented 2D corpus callosum data sets and 10 hippocampal 3D point-sets.

A Bayesian network formulation for relational shape matching is presented. The main advantage of the relational shape matching approach is the obviation of the nonrigid spatial mappings used by recent nonrigid matching approaches. The basic variables that need to be estimated in the relational shape matching objective function are the global rotation and scale and the local displacements and correspondences. The new Bethe free energy approach is used to estimate the pairwise correspondences between links of the template graphs and the data. The resulting framework is useful in both registration and recognition contexts. Results are shown on hand-drawn templates and on 2D transverse T1-weighted MR images.

The concave-convex procedure (CCCP) is a way to construct discrete-time iterative dynamical systems that are guaranteed to decrease global optimization and energy functions monotonically. This procedure can be applied to almost any optimization problem, and many existing algorithms can be interpreted in terms of it. In particular, we prove that all expectation-maximization algorithms and classes of Legendre minimization and variational bounding algorithms can be reexpressed in terms of CCCP. We show that many existing neural network and mean-field theory algorithms are also examples of CCCP. The generalized iterative scaling algorithm and Sinkhorn's algorithm can also be expressed as CCCP by changing variables. CCCP can be used both as a new way to understand, and prove the convergence of, existing optimization algorithms and as a procedure for generating new algorithms.

For patient setup verification in external beam radiotherapy (EBRT) of prostate cancer, we developed an information theoretic registration framework, called the minimax entropy registration framework, to simultaneously and iteratively segment portal images and register them to three-dimensional (3-D) computed tomography (CT) image data. The registration framework has two steps, the max step and the min step, and evaluates appropriate entropies to estimate segmentations of the portal images and to find the transformation parameters. In the initial version of the algorithm (Bansal et al. 1999), we assumed image pixels to be independently distributed, an assumption not true in general. Thus, to better segment the portal images and to improve the accuracy of the estimated registration parameters, in this initial formulation of the problem, the correlation among pixel intensities is modeled using a one-dimensional Markov random process. Line processes are incorporated into the model to improve the estimation of segmentation of the portal images. In the max step, the principle of maximum entropy is invoked to estimate the probability distribution on the segmentations. The estimated distribution is then incorporated into the min step to estimate the registration parameters. Performance of the proposed framework is evaluated and compared to that of a mutual information-based registration algorithm using both simulated and real patient data. In the proposed registration framework, registration of the 3-D CT image and the portal images is guided by an estimated segmentation of the pelvic bone. However, as the prostate can move with respect to the pelvic structure, further localization of the prostate using ultrasound image data is required, an issue to be further explored in future.

In a Bayesian tomographic maximum a posteriori (MAP) reconstruction, an estimate of the object f is computed by iteratively minimizing an objective function that typically comprises the sum of a log-likelihood (data consistency) term and prior (or penalty) term. The prior can be used to stabilize the solution and to also impose spatial properties on the solution. One such property, preservation of edges and locally monotonic regions, is captured by the well-known median root prior (MRP), an empirical method that has been applied to emission and transmission tomography. We propose an entirely new class of convex priors that depends on f and also on m, an auxiliary field in register with f. We specialize this class to our median prior (MP). The approximate action of the median prior is to draw, at each iteration, an object voxel toward its own local median. This action is similar to that of MRP and results in solutions that impose the same sorts of object properties as does MRP. Our MAP method is not empirical, since the problem is stated completely as the minimization of a joint (on f and m) objective. We propose an alternating algorithm to compute the joint MAP solution and apply this to emission tomography, showing that the reconstructions are qualitatively similar to those obtained using MRP.

We address the problem of Bayesian image reconstruction with a prior that captures the notion of a clustered intensity histogram. The problem is formulated in the framework of a joint-MAP (maximum a posteriori) estimation with the prior PDF modeled as a mixture-of-gammas density. This prior PDF has appealing properties, including positivity enforcement. The joint MAP optimization is carried out as an iterative alternating descent wherein a regularized likelihood estimate is followed by a mixture decomposition of the histogram of the current tomographic image estimate. The mixture decomposition step estimates the hyperparameters of the prior PDF. The objective functions associated with the joint MAP estimation are complicated and difficult to optimize, but we show how they may be transformed to allow for much easier optimization while preserving the fixed point of the iterations. We demonstrate the method in the context of medical emission and transmission tomography.

Deformable models are central to non-rigid motion analysis, shape matching and non-rigid medical image registration. Spline-based deformations are a very popular class of parameterizations of deformable models and have been heavily used in multiple domains. In a somewhat separate sub-field, weighted graphs are a frequently used object parameterization. Graph matching using weighted graph object parameterizations finds application in a spectrum ranging from rigid pose estimation to deformable object recognition. Here, we demonstrate a hitherto unsuspected relationship between spline-based deformable models and weighted graphs. It turns out that spline parameterizations in the kernel representation can be used to construct equivalent weighted graphs. With this connection established, we envision a cross-fertilization between these two seemingly disparate sub-fields of computer vision.

Bayesian belief propagation in graphical models has been recently shown to have very close ties to inference methods based in statis- tical physics. After Yedidia et al. demonstrated that belief prop- agation (cid:12)xed points correspond to extrema of the so-called Bethe free energy, Yuille derived a double loop algorithm that is guar- anteed to converge to a local minimum of the Bethe free energy. Yuille’s algorithm is based on a certain decomposition of the Bethe free energy and he mentions that other decompositions are possi- ble and may even be fruitful. In the present work, we begin with the Bethe free energy and show that it has a principled interpre- tation as pairwise mutual information minimization and marginal entropy maximization (MIME). Next, we construct a family of free energy functions from a spectrum of decompositions of the original Bethe free energy. For each free energy in this family, we develop a new algorithm that is guaranteed to converge to a local min- imum. Preliminary computer simulations are in agreement with this theoretical development.

We introduce the Concave-Convex procedure (CCCP) which con(cid:173) structs discrete time iterative dynamical systems which are guar(cid:173) anteed to monotonically decrease global optimization/energy func(cid:173) tions. It can be applied to (almost) any optimization problem and many existing algorithms can be interpreted in terms of CCCP. In particular, we prove relationships to some applications of Legendre transform techniques. We then illustrate CCCP by applications to Potts models, linear assignment, EM algorithms, and Generalized Iterative Scaling (GIS). CCCP can be used both as a new way to understand existing optimization algorithms and as a procedure for generating new algorithms.

We present a new robust point matching algorithm (RPM) that can jointly estimate the correspondence and non-rigid transformations between two point-sets that may be of different sizes. The algorithm utilizes the soft assign for the correspondence and the thin-plate spline for the non-rigid mapping. Embedded within a deterministic annealing framework, the algorithm can automatically reject a fraction of the points as outliers. Experiments on both 2D synthetic point-sets with varying degrees of deformation, noise and outliers, and on real 3D sulcal point-sets (extracted from brain MRI) demonstrate the robustness of the algorithm.

We propose a new heuristic for approximating the maximum clique problem based on a recently introduced deterministic annealing algorithm which generalizes Waugh and Westewelt's cluster-competitive net. The approach is centered around a fundamental result proved by Motzkin and Straus in the mid-1960s, and recently expanded in various ways, which allows us to formulate the maximum clique problem in terms of a linearly constrained quadratic program. Preliminary experiments on random as well as standard benchmark graphs are presented which demonstrate the validity of the approach.

We have developed a new mutual information-based registration method for matching unlabeled point features. In contrast to earlier mutual information-based registration methods, which estimate the mutual information using image intensity information, our approach uses the point feature location information. A novel aspect of our approach is the emergence of correspondence (between the two sets of features) as a natural by-product of joint density estimation. We have applied this algorithm to the problem of geometric alignment of primate autoradiographs. We also present preliminary results on three-dimensional robust matching of sulci derived from anatomical magnetic resonance images. Finally, we present an experimental comparison between the mutual information approach and other recent approaches which explicitly parameterize feature correspondence.

The softassign quadratic assignment algorithm is a discrete-time, continuous-state, synchronous updating optimizing neural network. While its effectiveness has been shown in the traveling salesman problem, graph matching, and graph partitioning in thousands of simulations, its convergence properties have not been studied. Here, we construct discrete-time Lyapunov functions for the cases of exact and approximate doubly stochastic constraint satisfaction, which show convergence to a fixed point. The combination of good convergence properties and experimental success makes the softassign algorithm an excellent choice for neural quadratic assignment optimization.

A fundamental open problem in computer vision—determining pose and correspondence between two sets of points in space—is solved with a novel, fast, robust and easily implementable algorithm. The technique works on noisy 2D or 3D point sets that may be of unequal sizes and may differ by non-rigid transformations. Using a combination of optimization techniques such as deterministic annealing and the softassign, which have recently emerged out of the recurrent neural network/statistical physics framework, analog objective functions describing the problems are minimized. Over thirty thousand experiments, on randomly generated points sets with varying amounts of noise and missing and spurious points, and on hand-written character sets demonstrate the robustness of the algorithm.

We present a novel method for the geometric alignment of autoradiographs of the brain. The method is based on finding the spatial mapping and the one-to-one correspondences (or homologies) between point features extracted from the images and rejecting non-homologies as outliers. In this way, we attempt to account for the local, natural and artifactual differences between the autoradiograph slices. We have used the resulting automated algorithm on a set of left prefrontal cortex autoradiograph slices, specifically demonstrated its ability to perform point outlier rejection, validated its robustness property using synthetically generated spatial mappings and provided an anecdotal visual comparison with the well-known iterated closest-point (ICP) algorithm. Visualization of a stack of aligned left prefrontal cortex autoradiograph slices is also provided.

A new algorithm for graph matching, which uses graduated assignment is presented, along with experimental results demonstrating large improvements in speed and accuracy over previous techniques. The softassign, a novel constraint satisfaction technique, is applied to a new graph matching energy function that uses a robust, sparse distance measure between the links of the two graphs. The softassign, which has emerged out of the neural network/statistical physics framework enforces two-way (assignment) constraints without the use of penalty terms. The algorithm's low order computational complexity [0(lm), where l and m are the number of links in the two graphs] compares favorably with most competing approaches. The method, not restricted to any special class of graph, is applied to subgraph isomorphism, weighted graph matching, and attributed relational graph matching. Experiments on graphs generated from images and on randomly generated graphs, including benchmarks against a relation labeling algorithm and an algorithm employing Potts glass dynamics are reported. Over twenty-five thousand experiments were conducted. No comparable results have been reported by any other graph matching algorithm before in the research literature.

We present a novel optimizing network architecture with applications in vision, learning, pattern recognition, and combinatorial optimization. This architecture is constructed by combining the following techniques: (1) deterministic annealing, (2) self-amplification, (3) algebraic transformations, (4) clocked objectives, and (5) softassign. Deterministic annealing in conjunction with self-amplification avoids poor local minima and ensures that a vertex of the hypercube is reached. Algebraic transformations and clocked objectives help partition the relaxation into distinct phases. The problems considered have doubly stochastic matrix constraints or minor variations thereof. We introduce a new technique, softassign, which is used to satisfy this constraint. Experimental results on different problems are presented and discussed.

Prior knowledge constraints are imposed upon a learning problem in the form of distance measures. Prototypical 2D point sets and graphs are learned by clustering with point-matching and graph-matching distance measures. The point-matching distance measure is approximately invariant under affine transformations—translation, rotation, scale, and shear—and permutations. It operates between noisy images with missing and spurious points. The graph-matching distance measure operates on weighted graphs and is invariant under permutations. Learning is formulated as an optimization problem. Large objectives so formulated (∼ million variables) are efficiently minimized using a combination of optimization techniques—softassign, algebraic transformations, clocked objectives, and deterministic annealing.

A graduated assignment algorithm for graph matching is presented which is fast and accurate even in the presence of high noise. By combining graduated nonconvexity, two-way (assignment) constraints, and sparsity, large improvements in accuracy and speed are achieved. Its low order computational complexity [O(lm), where l and m are the number of links in the two graphs] and robustness in the presence of noise offer advantages over traditional combinatorial approaches. The algorithm, not restricted to any special class of graph, is applied to subgraph isomorphism, weighted graph matching, and attributed relational graph matching. To illustrate the performance of the algorithm, attributed relational graphs derived from objects are matched. Then, results from twenty-five thousand experiments conducted on 100 mode random graphs of varying types (graphs with only zero-one links, weighted graphs, and graphs with node attributes and multiple link types) are reported. No comparable results have been reported by any other graph matching algorithm before in the research literature. Twenty-five hundred control experiments are conducted using a relaxation labeling algorithm and large improvements in accuracy are demonstrated.

A Lagrangian relaxation network for graph matching is presented. The problem is formulated as follows: given graphs G and g, find a permutation matrix M that brings the two sets of vertices into correspondence. Permutation matrix constraints are formulated in the framework of deterministic annealing. Our approach is in the same spirit as a Lagrangian decomposition approach in that the row and column constraints are satisfied separately with a Lagrange multiplier used to equate the two "solutions". Due to the unavoidable symmetries in graph isomorphism (resulting in multiple global minima), we add a symmetry-breaking self-amplification term in order to obtain a permutation matrix. With the application of a fixpoint preserving algebraic transformation to both the distance measure and self-amplification terms, we obtain a Lagrangian relaxation network. The network performs minimization with respect to the Lagrange parameters and maximization with respect to the permutation matrix variables. Simulation results are shown on 100 node random graphs and for a wide range of connectivities.

A geometric criterion is developed for establishing shape based non rigid correspondence between plane curves. Unlike previous efforts, the criterion does not use rigid invariants of shape. Instead, shapes are compared non rigidly from the vantage point of the correspondence. Geometric invariants are proposed for curves whose shapes can be exactly matched by a non rigid correspondence. The invariants are based on angular deviations of convex and concave segments of the curves. Examples of correspondences between curves obtained from medical images are provided.>

While the ML-EM algorithm for reconstruction for emission tomography is unstable due to the ill-posed nature of the problem. Bayesian reconstruction methods overcome this instability by introducing prior information, often in the form of a spatial smoothness regularizer. More elaborate forms of smoothness constraints may be used to extend the role of the prior beyond that of a stabilizer in order to capture actual spatial information about the object. Previously proposed forms of such prior distributions were based on the assumption of a piecewise constant source distribution. Here, the authors propose an extension to a piecewise linear model-the weak plate-which is more expressive than the piecewise constant model. The weak plate prior not only preserves edges but also allows for piecewise ramplike regions in the reconstruction. Indeed, for the authors' application in SPECT, such ramplike regions are observed in ground-truth source distributions in the form of primate autoradiographs of rCBF radionuclides. To incorporate the weak plate prior in a MAP approach, the authors model the prior as a Gibbs distribution and use a GEM formulation for the optimization. They compare quantitative performance of the ML-EM algorithm, a GEM algorithm with a prior favoring piecewise constant regions, and a GEM algorithm with their weak plate prior. Pointwise and regional bias and variance of ensemble image reconstructions are used as indications of image quality. The authors' results show that the weak plate and membrane priors exhibit improved bias and variance relative to ML-EM techniques.

This paper unifies "line-process" approaches for regularization with discontinuities and robust estimation techniques. We generalize the notion of a "line process" to that of an analog "outlier process" and show that a problem formulated in terms of outlier processes can be viewed in terms of robust statistics. We also characterize a class of robust statistical problems for which an equivalent outlier-process formulation exists and give a straightforward method for converting a robust estimation problem into an outlier-process formulation. This outlier-processes approach provides a general framework which subsumes the traditional line-process approaches as well as a wide class of robust estimation problems. Examples in image reconstruction and optical flow are used to illustrate the approach.>

A fundamental open problem in computer vision-determining pose and correspondence between two sets of points in space(cid:173) is solved with a novel, robust and easily implementable algorithm. The technique works on noisy point sets that may be of unequal sizes and may differ by non-rigid transformations. A 2D varia(cid:173) tion calculates the pose between point sets related by an affine transformation-translation, rotation, scale and shear. A 3D to 3D variation calculates translation and rotation. An objective describ(cid:173) ing the problem is derived from Mean field theory. The objective is minimized with clocked (EM-like) dynamics. Experiments with both handwritten and synthetic data provide empirical evidence for the method.

Prior constraints are imposed upon a learning problem in the form of distance measures. Prototypical 2-D point sets and graphs are learned by clustering with point matching and graph matching dis(cid:173) tance measures. The point matching distance measure is approx. invariant under affine transformations - translation, rotation, scale and shear - and permutations. It operates between noisy images with missing and spurious points. The graph matching distance measure operates on weighted graphs and is invariant under per(cid:173) mutations. Learning is formulated as an optimization problem . Large objectives so formulated ('" million variables) are efficiently minimized using a combination of optimization techniques - alge(cid:173) braic transformations, iterative projective scaling, clocked objec(cid:173) tives, and deterministic annealing.

Proposes a Bayesian method whereby maximum a posteriori (MAP) estimates of functional (PET and SPECT) images may be reconstructed with the aid of prior information derived from registered anatomical MR images of the same slice. The prior information consists of significant anatomical boundaries that are likely to correspond to discontinuities in an otherwise spatially smooth radionuclide distribution. The authors' algorithm, like others proposed recently, seeks smooth solutions with occasional discontinuities; the contribution here is the inclusion of a coupling term that influences the creation of discontinuities in the vicinity of the significant anatomical boundaries. Simulations on anatomically derived mathematical phantoms are presented. Although computationally intense in its current implication, the reconstructions are improved (ROI-RMS error) relative to filtered backprojection and EM-ML reconstructions. The simulations show that the inclusion of position-dependent anatomical prior Information leads to further improvement relative to Bayesian reconstructions without the anatomical prior. The algorithm exhibits a certain degree of robustness with respect to errors in the location of anatomical boundaries.

The authors are interested in solving the problems of image estimation and image segmentation in a joint maximum a posteriori (MAP) framework. Due to the computational complexity and non-convexity of the problem, a continuation method which tracks the minima through the variation of a control parameter is used. The authors have found it useful to define two new processes; the gradient (GRAD) and gradient-magnitude (GMAG) processes. The line process can be obtained through a monotonic transformation of the GMAG process. Interactions are still added in the line process domain, and the concept of the uncertainty function is introduced to characterize the properties of the GMAG-line process transformation. Results obtained using two different transformations are compared.>

An energy function for maximum a posteriori (MAP) image estimation is presented. The energy function is highly nonconvex, and finding the global minimum is a nontrival problem. When constraints on the interactions between line processes are removed, the deterministic, graduated nonconvexity (GNC) algorithm has been shown to find close to optimum solutions. The GNC model is generalized. Any number of constraints on the line processes can be added as a result of using the adiabatic approximation. The resulting algorithm is a combination of the conjugate gradient (CG) and the iterated conditional modes (ICM) algorithms and is completely deterministic. Since the GNC algorithm can be obtained as a special case of this approach, the algorithm is called the generalized GNC or G/sup 2/NC algorithm. It is executed on two aerial images. Results are presented along with comparisons to the GNC algorithm.>