Abstract In pathology, the spatial distribution and proportions of tissue types are key indicators of disease progression, and are more readily available than fine-grained annotations. However, these assessments are rarely mapped to pixel-wise segmentation. The task is fundamentally underdetermined, as many spatially distinct segmentations can satisfy the same global proportions in the absence of pixel-wise constraints. To address this, we introduce Variational Segmentation from Label Proportions (VSLP), a two-stage framework that infers dense segmentations from global label proportions, without any pixel-level annotations. This framework first leverages a pre-trained transformer model with test-time augmentation to produce a pixel-wise confidence estimate. In the second stage, these estimates are fused by solving a variational optimization problem that incorporates a Wasserstein data fidelity term alongside a learned regularizer. Unlike end-to-end networks, our variational method can visualize the fidelity–regularization energy, resulting in more interpretable segmentation. We validate our approach on two public datasets, achieving superior performance over existing weakly supervised and unsupervised methods. For one of these datasets, proportions have been estimated by an experienced pathologist to provide a realistic benchmark to the community. Furthermore, the method scales to an in-house dataset with noisy pathologist labels, severely outperforming state-of-the-art methods, thereby demonstrating practical applicability. The code and data will be made publicly available upon acceptance at https://github.com/xiaoliangpi/VSLP.

Abstract. Image restoration tasks often admit a wide range of solutions that are equally consistent with the observed data. Quantifying this uncertainty is crucial for the robust interpretation of restored images and their reliable use in science and decision-making. We consider the classification of images reconstructed from noisy and corrupted measurements, with special attention to the quantification of uncertainty in the delivered classification results. We address this problem by constructing a Bayesian statistical approach that combines learning-based image priors and image classifiers with explicit image observation models specified during inference. Following the manifold hypothesis, we assume that the image prior is supported on a submanifold of the ambient space—which we learn from uncorrupted training data using a variational autoencoder—and use as a classifier a support vector machine operating in this low-dimensional representation. The observation model is incorporated during inference time through its likelihood function. Bayesian computation is then efficiently performed by leveraging variants of the unadjusted Langevin algorithm that operate directly on the submanifold and are robust to multimodality. This results in a robust image classification method that provides uncertainty estimates that are provably well-posed, derived from Bayesian decision theory rigorously and transparently, and which incorporate physical and instrumental aspects of the data acquisition process through Bayes’ theorem. We demonstrate the effectiveness of the proposed approach through experiments on the MNIST and CelebA datasets, where we achieve accurate uncertainty estimates, as measured by the expected calibration error, even in challenging image restoration problems with significant inherent uncertainty.

In cardiac cine magnetic resonance imaging (MRI), the heart is repeatedly imaged at numerous time points during the cardiac cycle. Frequently, the temporal evolution of a certain region of interest such as the ventricles or the atria is highly relevant for clinical diagnosis. In this paper, we devise a novel approach that allows for an automatized propagation of an arbitrary region of interest (ROI) along the cardiac cycle from respective annotated ROIs provided by medical experts at two different points in time, most frequently at the end-systolic (ES) and the end-diastolic (ED) cardiac phases. At its core, a 3D TV-$\boldsymbol {L^{1}}$-based optical flow algorithm computes the apparent motion of consecutive MRI images in forward and backward directions. Subsequently, the given terminal annotated masks are propagated by this bidirectional optical flow in 3D, which results, however, in improper initial estimates of the segmentation masks due to numerical inaccuracies. These initially propagated segmentation masks are then refined by a 3D U-Net-based convolutional neural network (CNN), which was trained to enforce consistency with the forward and backward warped masks using a novel loss function. Moreover, a penalization term in the loss function controls large deviations from the initial segmentation masks. This method is benchmarked both on a new dataset with annotated single ventricles containing patients with severe heart diseases and on a publicly available dataset with different annotated ROIs. We emphasize that our novel loss function enables fine-tuning the CNN on a single patient, thereby yielding state-of-the-art results along the complete cardiac cycle.

Video denoising is a fundamental problem in numerous computer vision applications. State-of-the-art attention-based denoising methods typically yield good results, but require vast amounts of GPU memory and usually suffer from very long computation times. Especially in the field of restoring digitized high-resolution historic films, these techniques are not applicable in practice. To overcome these issues, we introduce a lightweight video denoising network that combines efficient axial-coronal-sagittal (ACS) convolutions with a novel shifted window attention formulation (ASwin), which is based on the memory-efficient aggregation of self- and cross-attention across video frames. We numerically validate the performance and efficiency of our approach on synthetic Gaussian noise. Moreover, we train our network as a general-purpose blind denoising model for real-world videos, using a realistic noise synthesis pipeline to generate clean-noisy video pairs. A user study and non-reference quality assessment prove that our method outperforms the state-of-the-art on real-world historic videos in terms of denoising performance and temporal consistency.

Various problems in computer vision and medical imaging can be cast as inverse problems. A frequent method for solving inverse problems is the variational approach, which amounts to minimizing an energy composed of a data fidelity term and a regularizer. Classically, handcrafted regularizers are used, which are commonly outperformed by state-of-the-art deep learning approaches. In this work, we combine the variational formulation of inverse problems with deep learning by introducing the data-driven general-purpose total deep variation regularizer. In its core, a convolutional neural network extracts local features on multiple scales and in successive blocks. This combination allows for a rigorous mathematical analysis including an optimal control formulation of the training problem in a mean-field setting and a stability analysis with respect to the initial values and the parameters of the regularizer. In addition, we experimentally verify the robustness against adversarial attacks and numerically derive upper bounds for the generalization error. Finally, we achieve state-of-the-art results for several imaging tasks.

Recent deep learning approaches focus on improving quantitative scores of dedicated benchmarks, and therefore only reduce the observation-related (aleatoric) uncertainty. However, the model-immanent (epistemic) uncertainty is less frequently systematically analyzed. In this work, we introduce a Bayesian variational framework to quantify the epistemic uncertainty. To this end, we solve the linear inverse problem of undersampled MRI reconstruction in a variational setting. The associated energy functional is composed of a data fidelity term and the total deep variation (TDV) as a learned parametric regularizer. To estimate the epistemic uncertainty we draw the parameters of the TDV regularizer from a multivariate Gaussian distribution, whose mean and covariance matrix are learned in a stochastic optimal control problem. In several numerical experiments, we demonstrate that our approach yields competitive results for undersampled MRI reconstruction. Moreover, we can accurately quantify the pixelwise epistemic uncertainty, which can serve radiologists as an additional resource to visualize reconstruction reliability.

We propose a novel learning-based framework for image reconstruction particularly designed for training without ground truth data, which has three major building blocks: energy-based learning, a patch-based Wasserstein loss functional, and shared prior learning. In energy-based learning, the parameters of an energy functional composed of a learned data fidelity term and a data-driven regularizer are computed in a mean-field optimal control problem. In the absence of ground truth data, we change the loss functional to a patch-based Wasserstein functional, in which local statistics of the output images are compared to uncorrupted reference patches. Finally, in shared prior learning, both aforementioned optimal control problems are optimized simultaneously with shared learned parameters of the regularizer to further enhance unsupervised image reconstruction. We derive several time discretization schemes of the gradient flow and verify their consistency in terms of Mosco convergence. In numerous numerical experiments, we demonstrate that the proposed method generates state-of-the-art results for various image reconstruction applications---even if no ground truth images are available for training.

Diverse inverse problems in imaging can be cast as variational problems composed of a task-specific data fidelity term and a regularization term. In this paper, we propose a novel learnable general-purpose regularizer exploiting recent architectural design patterns from deep learning. We cast the learning problem as a discrete sampled optimal control problem, for which we derive the adjoint state equations and an optimality condition. By exploiting the variational structure of our approach, we perform a sensitivity analysis with respect to the learned parameters obtained from different training datasets. Moreover, we carry out a nonlinear eigenfunction analysis, which reveals interesting properties of the learned regularizer. We show state-of-the-art performance for classical image restoration and medical image reconstruction problems.

Continuous image morphing is a classical task in image processing. The metamorphosis model proposed by Trouvé, Younes, and coworkers [M. I. Miller and L. Younes, Int. J. Comput. Vis., 41 (2001), pp. 61--84; A. Trouvé and L. Younes, Found. Comput. Math., 5 (2005), pp. 173--198] casts this problem in the frame of Riemannian geometry and geodesic paths between images. The associated metric in the space of images incorporates dissipation caused by a viscous flow transporting image intensities and its variations along motion paths. In many applications, images are maps from the image domain into a manifold (e.g., in diffusion tensor imaging (DTI), the manifold of symmetric positive definite matrices with a suitable Riemannian metric). In this paper, we propose a generalized metamorphosis model for manifold-valued images, where the range space is a finite-dimensional Hadamard manifold. A corresponding time discrete version was presented in [S. Neumayer, J. Persch, and G. Steidl, SIAM J. Imaging Sci., 11 (2018), pp. 1898--1930] based on the general variational time discretization proposed in [B. Berkels, A. Effland, and M. Rumpf, SIAM J. Imaging Sci., 8 (2015), pp. 1457--1488]. Here, we prove the Mosco--convergence of the time discrete metamorphosis functional to the proposed manifold-valued metamorphosis model, which implies the convergence of time discrete geodesic paths to a geodesic path in the (time continuous) metamorphosis model. In particular, the existence of geodesic paths is established. In particular, the existence of geodesic paths is established. In fact, images as maps into Hadamard manifold are not only relevant in applications, but it is also shown that the joint convexity of the distance function---which characterizes Hadamard manifolds---is a crucial ingredient to establish existence of the metamorphosis model.

The space of images can be equipped with a Riemannian metric measuring both the cost of transport of image intensities and the variation of image intensities along motion lines. The resulting metamorphosis model was introduced and analyzed in [M. I. Miller and L. Younes, Int. J. Comput. Vis., 41 (2001), pp. 61--84; A. Trouvé and L. Younes, Found. Comput. Math., 5 (2005), pp. 173--198], and a variational time discretization for the geodesic interpolation was proposed in [B. Berkels, A. Effland, and M. Rumpf, SIAM J. Imaging Sci., 8 (2015), pp. 1457--1488]. In this paper, this time discrete model is expanded and an image extrapolation via a discretization of the geometric exponential map is consistently derived for the variational time discretization. For a given weakly differentiable initial image and an initial image variation, the exponential map allows one to compute a discrete geodesic extrapolation path in the space of images. It is shown that a time step of this shooting method can be formulated in the associated deformations only. For sufficiently small time steps, local existence and uniqueness are proved using a suitable fixed point formulation and the implicit function theorem. A spatial Galerkin discretization with cubic splines on coarse meshes for the deformations and piecewise bilinear finite elements on fine meshes for the image intensities are used to derive a fully practical algorithm. Different applications underline the efficiency and stability of the proposed approach.

In this paper the space of images is considered as a Riemannian manifold using the metamorphosis approach (see [M. I. Miller and L. Younes, Int. J. Comput. Vis., 41 (2001), pp. 61--84; A. Trouvé and L. Younes, SIAM J. Math. Anal., 37 (2005), pp. 17--59; and A. Trouvé and L. Younes, Found. Comput. Math., 5 (2005), pp. 173--198]), where the underlying Riemannian metric simultaneously measures the cost of image transport and intensity variation. A robust and effective variational time discretization of geodesics paths is proposed. This requires minimizing a discrete path energy consisting of a sum of consecutive image matching functionals over a set of image intensity maps and pairwise matching deformations. For square-integrable input images the existence of discrete, connecting geodesic paths defined as minimizers of this variational problem is shown. Furthermore, $\Gamma$-convergence of the underlying discrete path energy to the continuous path energy is proved. This includes a diffeomorphism property for the induced transport and the existence of a square-integrable weak material derivative in space and time. A spatial discretization via finite elements combined with an alternating descent scheme in the set of image intensity maps and the set of matching deformations is presented to approximate discrete geodesic paths numerically. Computational results underline the efficiency of the proposed approach and demonstrate important qualitative properties.