We propose a method for test-time adaptation of pretrained depth completion models. Depth completion models, trained on some ``source'' data, often predict erroneous outputs when transferred to ``target'' data captured in novel environmental conditions due to a covariate shift. The crux of our method lies in quantifying the likelihood of depth predictions belonging to the source data distribution. The challenge is in the lack of access to out-of-distribution (target) data prior to deployment. Hence, rather than making assumptions regarding the target distribution, we utilize adversarial perturbations as a mechanism to explore the data space. This enables us to train an energy model that scores local regions of depth predictions as in- or out-of-distribution. We update the parameters of pretrained depth completion models at test time to minimize energy, effectively aligning test-time predictions to those of the source distribution. We call our method ``Energy-based Test-time Adaptation'', or ETA for short. We evaluate our method across three indoor and three outdoor datasets, where ETA improve over the previous state-of-the-art method by an average of 6.94% for outdoors and 10.23% for indoors. Project Page: https://fuzzythecat.github.io/eta.

We propose a model-agnostic, progressive test-time energy adaptation approach for medical image segmentation. Maintaining model performance across diverse medical datasets is challenging, as distribution shifts arise from inconsistent imaging protocols and patient variations. Unlike domain adaptation methods that require multiple passes through target data - impractical in clinical settings - our approach adapts pretrained models progressively as they process test data. Our method leverages a shape energy model trained on source data, which assigns an energy score at the patch level to segmentation maps: low energy represents in-distribution (accurate) shapes, while high energy signals out-of-distribution (erroneous) predictions. By minimizing this energy score at test time, we refine the segmentation model to align with the target distribution. To validate the effectiveness and adaptability, we evaluated our framework on eight public MRI (bSSFP, T1- and T2-weighted) and X-ray datasets spanning cardiac, spinal cord, and lung segmentation. We consistently outperform baselines both quantitatively and qualitatively.

We propose a method for metric-scale monocular depth estimation. Inferring depth from a single image is an ill-posed problem due to the loss of scale from perspective projection during the image formation process. Any scale chosen is a bias, typically stemming from training on a dataset; hence, existing works have instead opted to use relative (normalized, inverse) depth. Our goal is to recover metric-scaled depth maps through a linear transformation. The crux of our method lies in the observation that certain objects (e.g., cars, trees, street signs) are typically found or associated with certain types of scenes (e.g., outdoor). We explore whether language descriptions can be used to transform relative depth predictions to those in metric scale. Our method, RSA , takes as input a text caption describing objects present in an image and outputs the parameters of a linear transformation which can be applied globally to a relative depth map to yield metric-scaled depth predictions. We demonstrate our method on recent general-purpose monocular depth models on indoors (NYUv2, VOID) and outdoors (KITTI). When trained on multiple datasets, RSA can serve as a general alignment module in zero-shot settings. Our method improves over common practices in aligning relative to metric depth and results in predictions that are comparable to an upper bound of fitting relative depth to ground truth via a linear transformation. Code is available at: https://github.com/Adonis-galaxy/RSA.

In this paper, we investigate the design of escape mechanisms within the state-of-the-art decomposition-based evolutionary multi-objective Moea/d framework. We propose to track the number of improvements made with respect to the single-objective sub-problems defined by decomposition. This allows us to compute an estimated sub-problem improvement probability which serves as an activation signal for some solution perturbation mechanism to occur. We report the benefits of such an approach by conducting a comprehensive experimental analysis on a broad range of combinatorial bi-objective bit-string landscapes with variable dimensions and ruggedness. Our empirical findings provide evidence on the effectiveness of the proposed escape mechanism and its ability in providing substantial improvement over conventional Moea/d. Besides, we provide a detailed analysis of parameters impact and anytime behavior in order to better highlight the strength of the proposed techniques as a function of available budget and problem characteristics.

We present an image segmentation algorithm that is developed in an unsupervised deep learning framework. The delineation of object boundaries often fails due to the nuisance factors such as illumination changes and occlusions. Thus, we initially propose an unsupervised image decomposition algorithm to obtain an intrinsic representation that is robust with respect to undesirable bias fields based on a multiplicative image model. The obtained intrinsic image is subsequently provided to an unsupervised segmentation procedure that is developed based on a piecewise smooth model. The segmentation model is further designed to incorporate a geometric constraint imposed in the generative adversarial network framework where the discrepancy between the distribution of partitioning functions and the distribution of prior shapes is minimized. We demonstrate the effectiveness and robustness of the proposed algorithm in particular with bias fields and occlusions using simple yet illustrative synthetic examples and a benchmark dataset for image segmentation.

We present an adaptive regularization scheme for optimizing composite energy functionals arising in image analysis problems. The scheme automatically trades off data fidelity and regularization depending on the current data fit during the iterative optimization, so that regularization is strongest initially, and wanes as data fidelity improves, with the weight of the regularizer being minimized at convergence. We also introduce a Huber loss function in both data fidelity and regularization terms, and present an efficient convex optimization algorithm based on the alternating direction method of multipliers (ADMM) using the equivalent relation between the Huber function and the proximal operator of the one-norm. We illustrate and validate our adaptive Huber-Huber model on synthetic and real images in segmentation, motion estimation, and denoising problems.

We formulate an energy for segmentation that is designed to have preference for segmenting the coarse over fine structure of the image, without smoothing across boundaries of regions. The energy is formulated by integrating a continuum of scales from a scale space computed from the heat equation within regions. We show that the energy can be optimized without computing a continuum of scales, but instead from a single scale. This makes the method computationally efficient in comparison to energies using a discrete set of scales. We apply our method to texture and motion segmentation. Experiments on benchmark datasets show that a continuum of scales leads to better segmentation accuracy over discrete scales and other competing methods.

We present a shape descriptor based on integral kernels. Shape is represented in an implicit form and it is characterized by a series of isotropic kernels that provide desirable invariance properties. The shape features are characterized at multiple scales which form a signature that is a compact description of shape over a range of scales. The shape signature is designed to be invariant with respect to group transformations which include translation, rotation, scaling, and reflection. In addition, the integral kernels that characterize local shape geometry enable the shape signature to be robust with respect to undesirable perturbations while retaining discriminative power. Use of our shape signature is demonstrated for shape matching based on a number of synthetic and real examples.

We derive an easy-to-implement and efficient algorithm for solving multi-label image partitioning problems in the form of the problem addressed by Region Competition. These problems jointly determine a parameter for each of the regions in the partition. Given an estimate of the parameters, a fast approximate solution to the multi-label sub-problem is derived by a global update that uses smoothing and thresholding. The method is empirically validated to be robust to fine details of the image that plague local solutions. Further, in comparison to global methods for the multi-label problem, the method is more efficient and it is easy for a non-specialist to implement. We give sample Matlab code for the multi-label Chan-Vese problem in this paper. Experimental comparison to the state-of-the-art in multi-label solutions to Region Competition shows that our method achieves equal or better accuracy, with the main advantage being speed and ease of implementation.

We propose a method for tracking structures (e.g., ventricles and myocardium) in cardiac images (e.g., magnetic resonance) by propagating forward in time a previous estimate of the structures using a new physically motivated motion estimation scheme. Our method estimates motion by regularizing only within structures so that differing motions among different structures are not mixed. It simultaneously satisfies the physical constraints at the interface between a fluid and a medium that the normal component of the fluid's motion must match the normal component of the medium's motion and the No-Slip condition, which states that the tangential velocity approaches zero near the interface. We show that these conditions lead to partial differential equations with Robin boundary conditions at the interface, which couple the motion between structures. We show that propagating a segmentation across frames using our motion estimation scheme leads to more accurate segmentation than traditional motion estimation that does not use physical constraints. Our method is suited to interactive segmentation, prominently used in commercial applications for cardiac analysis, where segmentation propagation is used to predict a segmentation in the next frame. We show that our method leads to more accurate predictions than a popular and recent interactive method used in cardiac segmentation.

In this work, we address the multi-label Mumford-Shah problem, i.e., the problem of jointly estimating a partitioning of the domain of the image, and functions defined within regions of the partition. We create algorithms that are efficient, robust to undesirable local minima, and are easy-to-implement. Our algorithms are formulated by slightly modifying the underlying statistical model from which the multi-label Mumford-Shah functional is derived. The advantage of this statistical model is that the underlying variables: the labels and the functions are less coupled than in the original formulation, and the labels can be computed from the functions with more global updates. The resulting algorithms can be tuned to the desired level of locality of the solution: from fully global updates to more local updates. We demonstrate our algorithm on two applications: joint multi-label segmentation and denoising, and joint multi-label motion segmentation and flow estimation. We compare to the state-of-the-art in multi-label Mumford-Shah problems and show that we achieve more promising results.

This paper presents a novel method for the segmentation of regions of interest in mammograms. The algorithm concurrently delineates the boundaries of the breast boundary, the pectoral muscle, as well as dense regions that include candidate masses. The resulting representation constitutes an analysis of the global structure of the object in the mammogram. We propose a topographic representation called the isocontour map, in which a salient region forms a dense quasi-concentric pattern of contours. The topological and geometrical structure of the image is analyzed using an inclusion tree that is a hierarchical representation of the enclosure relationships between contours. The "saliency" of a region is measured topologically as the minimum nesting depth. Features at various scales are analyzed in multiscale isocontour maps, and we demonstrate that the multiscale scheme provides an efficient way of achieving better delineations. Experimental results demonstrate that the proposed method has potential as the basis for a prompting system in mammogram mass detection.

This paper examines the issue of scale in modeling texture for the purpose of segmentation. We propose a scale descriptor for texture and an energy minimization model to find the scale of a given texture at each location. For each pixel, we use the intensity distribution in a local patch around that pixel to determine the smallest size of the domain that can be used to generate neighboring patches. The energy functional we propose to minimize is comprised of three terms: The first is the dissimilarity measure using the Wasserstein distance or Kullback-Leibler divergence between neighboring patch distributions; the second maximizes the entropy of the local patch, and the third penalizes larger size at equal fidelity. Our experiments show the proposed scale model successfully captures the intrinsic scale of texture at each location. We also apply our scale descriptor for improving texture segmentation based on histogram matching [15].

Textures within real images vary in brightness, contrast, scale and skew as imaging conditions change. To enable recognition of textures in real images, it is necessary to employ a similarity measure which is invariant to these properties. Furthermore, since textures often appear on undulating surfaces, such invariances must necessarily be local rather than global. Despite these requirements, it is only relatively recently that texture recognition algorithms with local scale and affine invariance properties have begun to be reported. Typically, they comprise detecting feature points followed by geometric normalization prior to description. We describe a method based on invariant combinations of linear filters. Unlike previous methods, we introduce a novel family of filters, which provide scale invariance, resulting in a texture description invariant to local changes in orientation, contrast and scale and robust to local skew. Significantly, the family of filters enable local scale invariants to be defined without using a scale selection principle or a large number of filters. A texture discrimination method based on the A2 similarity measure applied to histograms derived from our filter responses outperforms existing methods for retrieval and classification results for both the Brodatz textures and the UIUC database, which has been designed to require local invariance.

In this work, we address the problem of segmenting multiple objects, under possible occlusions, in a level set framework. A variational energy that incorporates a piecewise constant representation of the image in terms of the object regions and the object spatial order is proposed. To resolve occluded boundaries, prior knowledge of the shape of objects is also introduced within the segmentation energy. By minimizing the above energy, we solve the segmentation with depth problem, i.e., estimating the object boundaries, the object intensities, and the spatial order. The segmentation with depth problem was originally dealt with by the Nitzberg–Mumford–Shiota (NMS) variational formulation, which proposes segmentation energies for each spatial order. We discuss the relationships and show the computational advantages of our formulation over the NMS model, mainly due to our treatment of spatial order estimation within a single energy. A novelty here is that the spatial order information available in the image model is used to dynamically impose prior shape constraints only to occluded boundaries. Also presented are experiments on synthetic and real images that have promising results.

This paper presents a shape representation and a variational framework for the construction of diffeomorphisms that establish "meaningful"correspondences between images, in that they preserve the local geometry of singularities such as region boundaries. At the same time, the shape representation allows enforcing shape information locally in determining such region boundaries. Our representation is based on a kernel descriptor that characterizes local shape. This shape descriptor is robust to noise and forms a scale-space in which an appropriate scale can be chosen depending on the size of features of interest in the scene. In order to preserve local shape during the matching procedure, we introduce a novel constraint to traditional energybased approaches to estimate diffeomorphic deformations, and enforce it in a variational framework.

For shapes represented as closed planar contours, we introduce a class of functionals which are invariant with respect to the Euclidean group and which are obtained by performing integral operations. While such integral invariants enjoy some of the desirable properties of their differential counterparts, such as locality of computation (which allows matching under occlusions) and uniqueness of representation (asymptotically), they do not exhibit the noise sensitivity associated with differential quantities and, therefore, do not require presmoothing of the input shape. Our formulation allows the analysis of shapes at multiple scales. Based on integral invariants, we define a notion of distance between shapes. The proposed distance measure can be computed efficiently and allows warping the shape boundaries onto each other; its computation results in optimal point correspondence as an intermediate step. Numerical results on shape matching demonstrate that this framework can match shapes despite the deformation of subparts, missing parts and noise. As a quantitative analysis, we report matching scores for shape retrieval from a database.

This paper presents a robust algorithm for the comparison of mammogram pairs. Salient regions are extracted in a topographic way. An integral invariant representation of shape, in combination with area and distance measures, are used for establishing their correspondences. The experimental results demonstrate that the algorithm can provide a useful tool for a ComputerAided Diagnosis system in mammography.