We present a comprehensive analysis of total variation (TV) on non-Euclidean domains and its eigenfunctions. We specifically address parameterized surfaces, a natural representation of the shapes used in 3D graphics. Our work sheds new light on the celebrated Beltrami and Anisotropic TV flows and explains experimental findings from recent years on shape spectral TV [Fumero et al. 2020 ] and adaptive anisotropic spectral TV [Biton and Gilboa 2022 ]. A new notion of convexity on surfaces is derived by characterizing structures that are stable throughout the TV flow, performed on surfaces. We establish and numerically demonstrate quantitative relationships between TV, area, eigenvalue, and eigenfunctions of the TV operator on surfaces. Moreover, we expand the shape spectral TV toolkit to include zero-homogeneous flows, leading to efficient and versatile shape processing methods. These methods are exemplified through applications in smoothing, enhancement, and exaggeration filters. We introduce a novel method that, for the first time, addresses the shape deformation task using TV. This deformation technique is characterized by the concentration of deformation along geometrical bottlenecks, shown to coincide with the discontinuities of eigenfunctions. Overall, our findings elucidate recent experimental observations in spectral TV, provide a diverse framework for shape filtering, and present the first TV-based approach to shape deformation.

We propose a learning framework based on stochastic Bregman iterations, also known as mirror descent, to train sparse neural networks with an inverse scale space approach. We derive a baseline algorithm called LinBreg, an accelerated version using momentum, and AdaBreg, which is a Bregmanized generalization of the Adam algorithm. In contrast to established methods for sparse training the proposed family of algorithms constitutes a regrowth strategy for neural networks that is solely optimization-based without additional heuristics. Our Bregman learning framework starts the training with very few initial parameters, successively adding only significant ones to obtain a sparse and expressive network. The proposed approach is extremely easy and efficient, yet supported by the rich mathematical theory of inverse scale space methods. We derive a statistically profound sparse parameter initialization strategy and provide a rigorous stochastic convergence analysis of the loss decay and additional convergence proofs in the convex regime. Using only $3.4\%$ of the parameters of ResNet-18 we achieve $90.2\%$ test accuracy on CIFAR-10, compared to $93.6\%$ using the dense network. Our algorithm also unveils an autoencoder architecture for a denoising task. The proposed framework also has a huge potential for integrating sparse backpropagation and resource-friendly training. Code is available at https://github.com/TimRoith/BregmanLearning.

The vulnerability of deep neural networks to small and even imperceptible perturbations has become a central topic in deep learning research. Although several sophisticated defense mechanisms have been introduced, most were later shown to be ineffective. However, a reliable evaluation of model robustness is mandatory for deployment in safety-critical scenarios. To overcome this problem we propose a simple yet effective modification to the gradient calculation of state-of-the-art first-order adversarial attacks. Normally, the gradient update of an attack is directly calculated for the given data point. This approach is sensitive to noise and small local optima of the loss function. Inspired by gradient sampling techniques from non-convex optimization, we propose Dynamically Sampled Nonlocal Gradient Descent (DSNGD). DSNGD calculates the gradient direction of the adversarial attack as the weighted average over past gradients of the optimization history. Moreover, distribution hyperparameters that define the sampling operation are automatically learned during the optimization scheme. We empirically show that by incorporating this nonlocal gradient information, we are able to give a more accurate estimation of the global descent direction on noisy and non-convex loss surfaces. In addition, we show that DSNGD-based attacks are on average 35% faster while achieving 0.9% to 27.1% higher success rates compared to their gradient descent-based counterparts.

The susceptibility of deep neural networks to untrustworthy predictions, including out-of-distribution (OOD) data and adversarial examples, still prevent their widespread use in safety-critical applications. Most existing methods either require a retraining of a given model to achieve robust identification of adversarial attacks or are limited to out-of-distribution sample detection only. In this work, we propose a geometric gradient analysis (GGA) to improve the identification of untrustworthy predictions without retraining of a given model. GGA analyzes the geometry of the loss landscape of neural networks based on the saliency maps of their respective input. We observe considerable differences between the input gradient geometry of trustworthy and untrustworthy predictions. Using these differences, GGA outperforms prior approaches in detecting OOD data and adversarial attacks, including state-of-the-art and adaptive attacks.

We present an adaptive regularization scheme for optimizing composite energy functionals arising in image analysis problems. The scheme automatically trades off data fidelity and regularization depending on the current data fit during the iterative optimization, so that regularization is strongest initially, and wanes as data fidelity improves, with the weight of the regularizer being minimized at convergence. We also introduce a Huber loss function in both data fidelity and regularization terms, and present an efficient convex optimization algorithm based on the alternating direction method of multipliers (ADMM) using the equivalent relation between the Huber function and the proximal operator of the one-norm. We illustrate and validate our adaptive Huber-Huber model on synthetic and real images in segmentation, motion estimation, and denoising problems.

The aim of this paper is to derive and analyze a variational model for the joint estimation of motion and reconstruction of image sequences, which is based on a time-continuous Eulerian motion model. The model can be set up in terms of the continuity equation or the brightness constancy equation. The analysis in this paper focuses on the latter for robust motion estimation on sequences of two-dimensional images. We rigorously prove the existence of a minimizer in a suitable function space setting. Moreover, we discuss the numerical solution of the model based on primal-dual algorithms and investigate several examples. Finally, the benefits of our model compared to existing techniques, such as sequential image reconstruction and motion estimation, are shown.

This paper discusses the use of absolutely one-homogeneous regularization functionals in a variational, scale space, and inverse scale space setting to define a nonlinear spectral decomposition of input data. We present several theoretical results that explain the relation between the different definitions. Additionally, results on the orthogonality of the decomposition, a Parseval-type identity, and the notion of generalized (nonlinear) eigenvectors closely link our nonlinear multiscale decompositions to the well-known linear filtering theory. Numerical results are used to illustrate our findings.

In this paper we present a novel iterative procedure for multichannel image and data reconstruction using Bregman distances. The motivation for our approach is that in many applications multiple channels share a common subgradient with respect to a suitable regularization. This implies desirable properties such as a common edge set (and a common direction of the normals to the level lines) in the case of the total variation (TV). Therefore, we propose to determine each iterate by regularizing each channel with a weighted linear combination of Bregman distances to all other image channels from the previous iteration. In this sense we generalize the Bregman iteration proposed by Osher et al. in [Multiscale Model. Simul., 4 (2005), pp. 460--489] to multichannel images. We prove the convergence of the proposed scheme, analyze stationary points, and present numerical experiments on color image denoising, which show the superior behavior of our approach in comparison to TV, TV with Bregman iterations on each channel separately, and vectorial TV. Further numerical experiments include image deblurring and image inpainting. Additionally, we propose using the infimal convolution of Bregman distances to different channels from the previous iteration to obtain the independence of the sign and hence the independence of the direction of the edge. While this work focuses on TV regularization, the proposed scheme can potentially improve any variational multichannel reconstruction method with a one-homogeneous regularization.

Earth-observing satellites usually not only take ordinary red-green-blue images but also provide several images including the near-infrared and infrared spectrum. These images are called multispectral, for about four to seven different bands, or hyperspectral, for higher dimensional images of up to 210 bands. The drawback of the additional spectral information is that each spectral band has rather low spatial resolution. In this paper we propose a new variational method for sharpening high dimensional spectral images with the help of a high resolution gray-scale image while preserving the spectral characteristics used for classification and identification tasks. We describe the application of split Bregman minimization to our energy, prove convergence speed, and compare the split Bregman method to a descent method based on the ideas of alternating directions minimization. Finally, we show results on Quickbird multispectral as well as on AVIRIS hyperspectral data.

Respiratory and cardiac motion leads to image degradation in positron emission tomography (PET) studies of the human heart. In this paper we present a novel approach to motion correction based on dual gating and mass-preserving hyperelastic image registration. Thereby, we account for intensity modulations caused by the highly nonrigid cardiac motion. This leads to accurate and realistic motion estimates which are quantitatively validated on software phantom data and carried over to clinically relevant data using a hardware phantom. For patient data, the proposed method is first evaluated in a high statistic (20 min scans) dual gating study of 21 patients. It is shown that the proposed approach properly corrects PET images for dual-cardiac as well as respiratory-motion. In a second study the list mode data of the same patients is cropped to a scan time reasonable for clinical practice (3 min). This low statistic study not only shows the clinical applicability of our method but also demonstrates its robustness against noise obtained by hyperelastic regularization.

Bregman methods introduced in [S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, Multiscale Model. Simul., 4 (2005), pp. 460–489] to image processing are demonstrated to be an efficient optimization method for solving sparse reconstruction with convex functionals, such as the $\ell^1$ norm and total variation [W. Yin, S. Osher, D. Goldfarb, and J. Darbon, SIAM J. Imaging Sci., 1 (2008), pp. 143–168; T. Goldstein and S. Osher, SIAM J. Imaging Sci., 2 (2009), pp. 323–343]. In particular, the efficiency of this method relies on the performance of inner solvers for the resulting subproblems. In this paper, we propose a general algorithm framework for inverse problem regularization with a single forward-backward operator splitting step [P. L. Combettes and V. R. Wajs, Multiscale Model. Simul., 4 (2005), pp. 1168–1200], which is used to solve the subproblems of the Bregman iteration. We prove that the proposed algorithm, namely, Bregmanized operator splitting (BOS), converges without fully solving the subproblems. Furthermore, we apply the BOS algorithm and a preconditioned one for solving inverse problems with nonlocal functionals. Our numerical results on deconvolution and compressive sensing illustrate the performance of nonlocal total variation regularization under the proposed algorithm framework, compared to other regularization techniques such as the standard total variation method and the wavelet-based regularization method.

The Cahn–Hilliard equation is a nonlinear fourth order diffusion equation originating in material science for modeling phase separation and phase coarsening in binary alloys. The inpainting of binary images using the Cahn–Hilliard equation is a new approach in image processing. In this paper we discuss the stationary state of the proposed model and introduce a generalization for grayvalue images of bounded variation. This is realized by using subgradients of the total variation functional within the flow, which leads to structure inpainting with smooth curvature of level sets.

Defocus can be modeled as a diffusion process and represented mathematically using the heat equation, where image blur corresponds to the diffusion of heat. This analogy can be extended to non-planar scenes by allowing a space-varying diffusion coefficient. The inverse problem of reconstructing 3-D structure from blurred images corresponds to an "inverse diffusion" that is notoriously ill-posed. We show how to bypass this problem by using the notion of relative blur. Given two images, within each neighborhood, the amount of diffusion necessary to transform the sharper image into the blurrier one depends on the depth of the scene. This can be used to devise a global algorithm to estimate the depth profile of the scene without recovering the deblurred image, using only forward diffusion.