The rapid advancement of generative models, particularly diffusion-based methods, has significantly improved the realism of synthetic images. As new generative models continuously emerge, detecting generated images remains a critical challenge. While fully supervised, and few-shot methods have been proposed, maintaining an updated dataset is time-consuming and challenging. Consequently, zero-shot methods have gained increasing attention in recent years. We find that existing zero-shot methods often struggle to adapt to specific image domains, such as artistic images, limiting their real-world applicability. In this work, we introduce CLIDE, a novel zero-shot detection method based on conditional likelihood approximation. Our approach computes likelihoods conditioned on real images, enabling adaptation across diverse image domains. We extensively evaluate CLIDE, demonstrating SOTA performance on a large-scale general dataset and significantly outperform existing methods in domain-specific cases. These results demonstrate the robustness of our method and underscore the need of broad, domain-aware generalization for the AI-generated image detection task. Code is available at https://tinyurl.com/clide-detector.

Abstract We propose a new way to explain and to visualize neural network classification through a decomposition-based explainable AI (DXAI). Instead of providing an explanation heatmap, our method yields a decomposition of the image into class-agnostic and class-distinct parts, with respect to the data and chosen classifier. Following a fundamental signal processing paradigm of analysis and synthesis, the original image is the sum of the decomposed parts. We thus obtain a radically different way of explaining classification. The class-agnostic part ideally is composed of all image features which do not possess class information, where the class-distinct part is its complementary. This new visualization can be more helpful and informative in certain scenarios, especially when the attributes are dense, global, and additive in nature, for instance, when colors or textures are essential for class distinction. Code is available at https://github.com/dxai2024/dxai .

The ability to cope with out-of-distribution (OOD) corruptions and adversarial attacks is crucial in real-world safety-demanding applications. In this study, we develop a general mechanism to increase point clouds neural networks robustness based on focus analysis. Recent studies have revealed the phenomenon of Overfocusing, which leads to a performance drop. When the network is primarily influenced by small input regions, it becomes less robust and prone to misclassify under noise and corruptions. However, quantifying overfocusing is still vague and lacks clear definitions. Here, we provide a mathematical definition of focus, overfocusing and underfocusing. The notions are general, but in this study, we specifically investigate the case of 3D point clouds. We observe that corrupted sets result in a biased focus distribution compared to the clean training set. We show that as focus distribution deviates from the one learned in the training phase - classification performance deteriorates. We thus propose a parameter-free refocusing algorithm that aims to unify all corruptions under the same distribution. We validate our findings on a 3D zero-shot classification task, achieving SOTA in robust 3D classification on ModelNet-C dataset, and in adversarial defense against Shape-Invariant attack.

Distinguishing between real and AI-generated images, commonly referred to as 'image detection', presents a timely and significant challenge. Despite extensive research in the (semi-)supervised regime, zero-shot and few-shot solutions have only recently emerged as promising alternatives. Their main advantage is in alleviating the ongoing data maintenance, which quickly becomes outdated due to advances in generative technologies. We identify two main gaps: (1) a lack of theoretical grounding for the methods, and (2) significant room for performance improvements in zero-shot and few-shot regimes. Our approach is founded on understanding and quantifying the biases inherent in generated content, where we use these quantities as criteria for characterizing generated images. Specifically, we explore the biases of the implicit probability manifold, captured by a pre-trained diffusion model. Through score-function analysis, we approximate the curvature, gradient, and bias towards points on the probability manifold, establishing criteria for detection in the zero-shot regime. We further extend our contribution to the few-shot setting by employing a mixture-of-experts methodology. Empirical results across 20 generative models demonstrate that our method outperforms current approaches in both zero-shot and few-shot settings. This work advances the theoretical understanding and practical usage of generated content biases through the lens of manifold analysis.

Likelihood approximations for images are not trivial to compute and can be useful in many applications. We examine the use of Contrastive Language-Image Pre-training (CLIP) to assess the likelihood of images and captions. We introduce Whitened CLIP, a novel transformation of the CLIP latent space via an invertible linear operation. This transformation ensures that each feature in the embedding space has zero mean, unit standard deviation, and no correlation with all other features, resulting in an identity covariance matrix. We show that the whitened embedding statistics can be well approximated by a standard normal distribution, allowing log-likelihood to be estimated using the squared Euclidean norm in the whitened space. The whitening procedure is completely training-free and uses a precomputed whitening matrix, making it extremely fast. We present several preliminary experiments demonstrating the properties and applicability of these likelihood scores to images and captions. Our code is available at github.com/rbetser/W_CLIP/tree/main.

Contrastive Language-Image Pre-Training (CLIP) is highly instrumental in machine learning applications within a large variety of domains. We investigate the geometry of this embedding, which is still not well understood, and show that text and image reside on linearly separable ellipsoid shells, not centered at the origin. We explain the benefits of having this structure, allowing to better embed instances according to their uncertainty during contrastive training. Frequent concepts in the dataset yield more false negatives, inducing greater uncertainty. A new notion of conformity is introduced, which measures the average cosine similarity of an instance to any other instance within a representative data set. We show this measure can be accurately estimated by simply computing the cosine similarity to the modality mean vector. Furthermore, we find that CLIP's modality gap optimizes the matching of the conformity distributions of image and text.

As neural networks grow in scale, their training becomes both computationally demanding and rich in dynamics. Amidst the flourishing interest in these training dynamics, we present a novel observation: Parameters during training exhibit intrinsic correlations over time. Capitalizing on this, we introduce \emph{correlation mode decomposition} (CMD). This algorithm clusters the parameter space into groups, termed modes, that display synchronized behavior across epochs. This enables CMD to efficiently represent the training dynamics of complex networks, like ResNets and Transformers, using only a few modes. Moreover, test set generalization is enhanced. We introduce an efficient CMD variant, designed to run concurrently with training. Our experiments indicate that CMD surpasses the state-of-the-art method for compactly modeled dynamics on image classification. Our modeling can improve training efficiency and lower communication overhead, as shown by our preliminary experiments in the context of federated learning.

We present a comprehensive analysis of total variation (TV) on non-Euclidean domains and its eigenfunctions. We specifically address parameterized surfaces, a natural representation of the shapes used in 3D graphics. Our work sheds new light on the celebrated Beltrami and Anisotropic TV flows and explains experimental findings from recent years on shape spectral TV [Fumero et al. 2020 ] and adaptive anisotropic spectral TV [Biton and Gilboa 2022 ]. A new notion of convexity on surfaces is derived by characterizing structures that are stable throughout the TV flow, performed on surfaces. We establish and numerically demonstrate quantitative relationships between TV, area, eigenvalue, and eigenfunctions of the TV operator on surfaces. Moreover, we expand the shape spectral TV toolkit to include zero-homogeneous flows, leading to efficient and versatile shape processing methods. These methods are exemplified through applications in smoothing, enhancement, and exaggeration filters. We introduce a novel method that, for the first time, addresses the shape deformation task using TV. This deformation technique is characterized by the concentration of deformation along geometrical bottlenecks, shown to coincide with the discontinuities of eigenfunctions. Overall, our findings elucidate recent experimental observations in spectral TV, provide a diverse framework for shape filtering, and present the first TV-based approach to shape deformation.

Graph is a highly generic and diverse representation, suitable for almost any data processing problem. Spectral graph theory has been shown to provide powerful algorithms, backed by solid linear algebra theory. It thus can be extremely instrumental to design deep network building blocks with spectral graph characteristics. For instance, such a network allows the design of optimal graphs for certain tasks or obtaining a canonical orthogonal low-dimensional embedding of the data. Recent attempts to solve this problem were based on minimizing Rayleigh-quotient type losses. We propose a different approach of directly learning the graph's eigensapce. A severe problem of the direct approach, applied in batch-learning, is the inconsistent mapping of features to eigenspace coordinates in different batches. We analyze the degrees of freedom of learning this task using batches and propose a stable alignment mechanism that can work both with batch changes and with graph-metric changes. We show that our learnt spectral embedding is better in terms of NMI, ACC, Grassman distnace, orthogonality and classification accuracy, compared to SOTA. In addition, the learning is more stable.

Robust point cloud classification is crucial for real-world applications, as consumer-type 3D sensors often yield partial and noisy data, degraded by various artifacts. In this work we propose a general ensemble framework, based on partial point cloud sampling. Each ensemble member is exposed to only partial input data. Three sampling strategies are used jointly, two local ones, based on patches and curves, and a global one of random sampling.We demonstrate the robustness of our method to various local and global degradations. We show that our framework significantly improves the robustness of top classification netowrks by a large margin. Our experimental setting uses the recently introduced ModelNet-C database by Ren et al. [24], where we reach SOTA both on unaugmented and on augmented data. Our unaugmented mean Corruption Error (mCE) is 0.64 (current SOTA is 0.86) and 0.50 for augmented data (current SOTA is 0.57). We analyze and explain these remarkable results through diversity analysis. Our code is availabe at: https://github.com/yossilevii100/EPiC

Neural networks have revolutionized the field of data science, yielding remarkable solutions in a data-driven manner. For instance, in the field of mathematical imaging, they have surpassed traditional methods based on convex regularization. However, a fundamental theory supporting the practical applications is still in the early stages of development. We take a fresh look at neural networks and examine them via nonlinear eigenvalue analysis. The field of nonlinear spectral theory is still emerging, providing insights about nonlinear operators and systems. In this paper we view a neural network as a complex nonlinear operator and attempt to find its nonlinear eigenvectors. We first discuss the existence of such eigenvectors and analyze the kernel of ReLU networks. Then we study a nonlinear power method for generic nonlinear operators. For proximal operators associated to absolutely one-homogeneous convex regularization functionals, we can prove convergence of the method to an eigenvector of the proximal operator. This motivates us to apply a nonlinear method to networks which are trained to act similarly as a proximal operator. In order to take the non-homogeneity of neural networks into account we define a modified version of the power method. We perform extensive experiments for different proximal operators and on various shallow and deep neural networks designed for image denoising. Proximal eigenvectors will be used for geometric analysis of graphs, as clustering or the computation of distance functions. For simple neural nets, we observe the influence of training data on the eigenvectors. For state-of-the-art denoising networks, we show that eigenvectors can be interpreted as (un)stable modes of the network, when contaminated with noise or other degradations.

Finding latent structures in data is drawing increasing attention in diverse fields such as image and signal processing, fluid dynamics, and machine learning. In this work we examine the problem of finding the main modes of gradient flows. Gradient descent is a fundamental process in optimization where its stochastic version is prominent in training of neural networks. Here our aim is to establish a consistent theory for gradient flows $\boldsymbol{\psi}_t = P(\boldsymbol{\psi})$, where $P$ is a nonlinear homogeneous operator. Our proposed framework stems from analytic solutions of homogeneous flows, previously formalized by Cohen and Gilboa, where the initial condition $\boldmath{\psi}_0$ admits the nonlinear eigenvalue problem $P(\boldsymbol{\psi}_0)=\lambda \boldsymbol{\psi}_0 $. We first present an analytic solution for dynamic mode decomposition (DMD) in such cases. We show an inherent flaw of DMD, which is unable to recover the essential dynamics of the flow. It is evident that DMD is best suited for homogeneous flows of degree one. We propose an adaptive time sampling scheme and show its dynamics are analogue to homogeneous flows of degree one with a fixed step size. Moreover, we adapt DMD to yield a real spectrum, using symmetric matrices. Our analytic solution of the proposed scheme recovers the dynamics perfectly and yields zero error. We then proceed to show the relation between the orthogonal modes $\{\phi_i\}$ and their decay profiles under the gradient flow. We formulate orthogonal nonlinear spectral decomposition (OrthoNS), which recovers the essential latent structures of the gradient descent process. Definitions for spectrum and filtering are given, and a Parseval-type identity is shown. Experimental results on images show the resemblance to direct computations of nonlinear spectral decomposition. A significant speedup (by about two orders of magnitude) is achieved for this application using the proposed method.

This work considers the problem of depth completion, with or without image data, where an algorithm may measure the depth of a prescribed limited number of pixels. The algorithmic challenge is to choose pixel positions strategically and dynamically to maximally reduce overall depth estimation error. This setting is realized in daytime or nighttime depth completion for autonomous vehicles with a programmable LiDAR. Our method uses an ensemble of predictors to define a sampling probability over pixels. This probability is proportional to the variance of the predictions of ensemble members, thus highlighting pixels that are difficult to predict. By additionally proceeding in several prediction phases, we effectively reduce redundant sampling of similar pixels. Our ensemble-based method may be implemented using any depth-completion learning algorithm, such as a state-of-the-art neural network, treated as a black box. In particular, we also present a simple and effective Random Forest-based algorithm, and similarly use its internal ensemble in our design. We conduct experiments on the KITTI dataset, using the neural network algorithm of Ma et al. and our Random Forest-based learner for implementing our method. The accuracy of both implementations exceeds the state of the art. Compared with a random or grid sampling pattern, our method allows a reduction by a factor of 4-10 in the number of measurements required to attain the same accuracy.

Depth acquisition, based on active illumination, is essential for autonomous and robotic navigation. LiDARs (Light Detection And Ranging) with mechanical, fixed, sampling templates are commonly used in today's autonomous vehicles. An emerging technology, based on solid-state depth sensors, with no mechanical parts, allows fast and adaptive scans. In this paper, we propose an adaptive, image-driven, fast, sampling and reconstruction strategy. First, we formulate a piece-wise planar depth model and estimate its validity for indoor and outdoor scenes. Our model and experiments predict that, in the optimal case, adaptive sampling strategies with about 20-60 piece-wise planar structures can approximate well a depth map. This translates to requiring a single depth sample for every 1200 RGB samples (less than 0.1%), providing strong motivation to investigate an adaptive framework. Second, we introduce SPS (Super-Pixel Sampler), a simple, generic, sampling and reconstruction algorithm, based on super-pixels. Our sampling improves grid and random sampling, consistently, for a wide variety of reconstruction methods. Third, we propose an extremely simple and fast reconstruction for our sampler. It achieves state-of-the-art results, compared to complex image- guided depth completion algorithms, reducing the required sampling rate by a factor of 3-4. A single-pixel prototype sampler built in our lab illustrates the concept.

Non-linear spectral decompositions of images based on one-homogeneous functionals such as total variation have gained considerable attention in the last few years. Due to their ability to extract spectral components corresponding to objects of different size and contrast, such decompositions enable filtering, feature transfer, image fusion and other applications. However, obtaining this decomposition involves solving multiple non-smooth optimisation problems and is therefore computationally highly intensive. In this paper, we present a neural network approximation of a non-linear spectral decomposition. We report up to four orders of magnitude (×10,000) speedup in processing of mega-pixel size images, compared to classical GPU implementations. Our proposed network, TVspecNET, is able to implicitly learn the underlying PDE and, despite being entirely data driven, inherits invariances of the model based transform. To the best of our knowledge, this is the first approach towards learning a non-linear spectral decomposition of images. Not only do we gain a staggering computational advantage, but this approach can also be seen as a step towards studying neural networks that can decompose an image into spectral components defined by a user rather than a handcrafted functional.

We propose a unified framework for isolating, comparing and differentiating objects within an image. We rely on the recently proposed total-variation transform, yielding a continuous, multi-scale, fully edge-preserving, local descriptor, referred to as spectral total-variation local scale signatures. We show and analyze several useful merits of this framework. Signatures are sensitive to size, local contrast and composition of structures; are invariant to translation, rotation, flip and linear illumination changes; and texture signatures are robust to the underlying structures. We prove exact conditions in the 1D case. We propose several applications for this framework: saliency map extraction for fusion of thermal and optical images or for medical imaging, clustering of vein-like features and size-based image manipulation.

Nonlinear eigenfunctions, induced by subgradients of one-homogeneous functionals (such as the 1-Laplacian), have shown to be instrumental in segmentation, clustering, and image decomposition. We present a class of flows for finding such eigenfunctions, generalizing a method recently suggested by Nossek and Gilboa. We analyze the flows on grids and graphs in the time-continuous and time-discrete settings. For a specific type of flow within this class, we prove convergence of the numerical iterations procedure and prove existence and uniqueness of the time-continuous case. Several toy examples are provided for illustrating the theoretical results, showing how such flows can be used on images and graphs.

The color of an object imaged in a turbid medium varies with distance and medium properties, deeming color an unstable source of information. Assuming 3D scene structure has become relatively easy to estimate, the main challenge in color recovery is calibrating medium properties in situ, at the time of acquisition. Existing attenuation calibration methods use either color charts, external hardware, or multiple images of an object. Here we show none of these is needed for calibration. We suggest a method for estimating the medium properties (both attenuation and scattering) using only images of backscattered light from the system's light sources. This is advantageous in turbid media where the object signal is noisy, and also alleviates the need for correspondence matching, which can be difficult in high turbidity. We demonstrate the advantages of our method through simulations and in a real-life experiment at sea.

We propose a new way to solve a very general blind inverse problem of multiple simultaneous degradations, such as blur, resolution reduction, noise, and contrast changes, without explicitly estimating the degradation. The proposed concept is based on combining semantic non-rigid patches, problem-specific high-quality prior data, and non-rigid registration tools. We show how a significant quality enhancement can be achieved, both visually and quantitatively, in the case of facial images. The method is demonstrated on the problem of cellular photography quality enhancement of dark facial images for different identities, expressions, and poses, and is compared with the state-of-the-art denoising, deblurring, super-resolution, and color-correction methods.

The revolution of RGB-D sensors is advancing towards mobile platforms for robotics, autonomous vehicles and consumer hand-helddevices. Strong pressures on power consumption and system price requirenew powerful algorithms that can robustly handle very low quality rawdata. In this paper we demonstrate the ability to reliably recover depth measurements from a variety of highly degraded depth modalities, coupled with standard RGB imagery. The method is based on a regularizer which fuses super-pixel information with the total-generalized-variation (TGV) functional. We examine our algorithm on several different degradations, includingnew Intel's RealSense hand-held device, LiDAR-type data and ultra-sparse random sampling. In all modalities which are heavily degraded, our robust algorithm achieves superior performance over the state-ofthe-art. Additionally, a robust error measure based on Tukey's biweight metricis suggested, which is better at ranking algorithm performance since itdoes not reward blurry non-physical depth results.

Depth restoration, the task of correcting depth noise and artifacts, has recently risen in popularity due to the increase in commodity depth cameras. When assessing the quality of existing methods, most researchers resort to the popular Middlebury dataset, however, this dataset was not created for depth enhancement, and therefore lacks the option of comparing genuine low-quality depth images with their high-quality, ground-truth counterparts. To address this shortcoming, we present the Depth Restoration Occlusionless Temporal (DROT) dataset. This dataset offers real depth sensor input coupled with registered pixel-to-pixel color images, and the ground-truth depth to which we wish to compare. Our dataset includes not only Kinect 1 and Kinect 2 data, but also an Intel R200 sensor intended for integration into hand-held devices. Beyond this, we present a new temporal depth-restoration method. Utilizing multiple frames, we create a number of possibilities for an initial degraded depth map, which allows us to arrive at a more educated decision when refining depth images. Evaluating this method with our dataset shows significant benefits, particularly for overcoming real sensor-noise artifacts.

This paper discusses the use of absolutely one-homogeneous regularization functionals in a variational, scale space, and inverse scale space setting to define a nonlinear spectral decomposition of input data. We present several theoretical results that explain the relation between the different definitions. Additionally, results on the orthogonality of the decomposition, a Parseval-type identity, and the notion of generalized (nonlinear) eigenvectors closely link our nonlinear multiscale decompositions to the well-known linear filtering theory. Numerical results are used to illustrate our findings.

In this paper, we introduce a novel notion of separation surfaces for image decomposition. A surface is embedded in the spectral total-variation (TV) 3D domain and encodes a spatially varying separation scale. The method allows good separation of textures with gradually varying pattern size, pattern contrast, or illumination. The recently proposed TV spectral framework is used to decompose the image into a continuum of textural scales. A desired texture, within a scale range, is found by fitting a surface to the local maximal responses in the spectral domain. A band above and below the surface, referred to as the texture stratum, defines for each pixel the adaptive scale range of the texture. Based on the decomposition, an application is proposed, which can attenuate or enhance textures in the image in a very natural and visually convincing manner.

This paper presents the idea of learning optimal filters for color image reconstruction based on a novel concept of nonlinear spectral image decompositions recently proposed by Guy Gilboa. We use a multiscale image decomposition approach based on total variation regularization and Bregman iterations to represent the input data as the sum of image layers containing features at different scales. Filtered images can be obtained by weighted linear combinations of the different frequency layers. We introduce the idea of learning optimal filters for the task of image denoising, and propose the idea of mixing high frequency components of different color channels. Our numerical experiments demonstrate that learning the optimal weights can significantly improve the results in comparison to the standard variational approach, and achieves state-of-the-art image denoising results.

Curvature is a fundamental component in differential geometry. It is used extensively in signal, image and shape processing, as a feature and in segmentation flows and regularization processes. In this paper we extend the notion of curvature in two ways. First, we present the Menger curvature which goes beyond classical curves and Riemannian manifolds to general metric spaces and is rigorously defined on a variety of discrete settings. We further extend the curvature to become a non-local entity using an adaptive, non-local integration measure, allowing curvature to be computed in a robust manner. Examples on natural and textural images highlight potential applications of these new concepts.

We examine the problem of matching patches of internal and external data. A new patch quality indicator is proposed which aims at maximizing the ratio of the number of points of interest to the area. We prove that the best convex polygon maximizing the indicator is the triangle. Thus data-driven triangle patches are used in the patch-matching algorithm, considerably improving the matching quality. The triangle representation inherently allows affine correction of the patches, thereby increasing the matching probability and enabling a smaller external data-set. We employ this technique for image quality enhancement of low resolution facial data using a data bank of external faces.

A new total variation (TV) spectral framework is presented. A TV transform is proposed which can be interpreted as a spectral domain, where elementary TV features, such as disks, approach impulses. A reconstruction formula from the spectral to the spatial domain is given, allowing the design of new filters. The framework formulates a new representation of images which can enhance the understanding of scales in the $L^1$ sense and improve the analysis and processing of textures. An example of a texture processing application illustrates possible benefits of this new framework.

A general scale space algorithm is presented for denoising signals and images with spatially varying dominant scales. The process is formulated as a partial differential equation with spatially varying time. The proposed adaptivity is semi-local and is in conjunction with the classical gradient-based diffusion coefficient, designed to preserve edges. The new algorithm aims at maximizing a local SNR measure of the denoised image. It is based on a generalization of a global stopping time criterion presented recently by the author and colleagues. Most notably, the method works well also for partially textured images and outperforms any selection of a global stopping time. Given an estimate of the noise variance, the procedure is automatic and can be applied well to most natural images.

This paper is concerned with finding the best partial differential equation-based denoising process, out of a set of possible ones. We focus on either finding the proper weight of the fidelity term in the energy minimization formulation or on determining the optimal stopping time of a nonlinear diffusion process. A necessary condition for achieving maximal SNR is stated, based on the covariance of the noise and the residual part. We provide two practical alternatives for estimating this condition by observing that the filtering of the image and the noise can be approximated by a decoupling technique, with respect to the weight or time parameters. Our automatic algorithm obtains quite accurate results on a variety of synthetic and natural images, including piecewise smooth and textured ones. We assume that the statistics of the noise were previously estimated. No a priori knowledge regarding the characteristics of the clean image is required. A theoretical analysis is carried out, where several SNR performance bounds are established for the optimal strategy and for a widely used method, wherein the variance of the residual part equals the variance of the noise.

Denoising algorithms based on gradient dependent regularizers, such as nonlinear diffusion processes and total variation denoising, modify images towards piecewise constant functions. Although edge sharpness and location is well preserved, important information, encoded in image features like textures or certain details, is often compromised in the process of denoising. We propose a mechanism that better preserves fine scale features in such denoising processes. A basic pyramidal structure-texture decomposition of images is presented and analyzed. A first level of this pyramid is used to isolate the noise and the relevant texture components in order to compute spatially varying constraints based on local variance measures. A variational formulation with a spatially varying fidelity term controls the extent of denoising over image regions. Our results show visual improvement as well as an increase in the signal-to-noise ratio over scalar fidelity term processes. This type of processing can be used for a variety of tasks in partial differential equation-based image processing and computer vision, and is stable and meaningful from a mathematical viewpoint.

The linear and nonlinear scale spaces, generated by the inherently real-valued diffusion equation, are generalized to complex diffusion processes, by incorporating the free Schrödinger equation. A fundamental solution for the linear case of the complex diffusion equation is developed. Analysis of its behavior shows that the generalized diffusion process combines properties of both forward and inverse diffusion. We prove that the imaginary part is a smoothed second derivative, scaled by time, when the complex diffusion coefficient approaches the real axis. Based on this observation, we develop two examples of nonlinear complex processes, useful in image processing: a regularized shock filter for image enhancement and a ramp preserving denoising process.

The widely used denoising algorithms based on nonlinear diffusion, such as Perona-Malik and total variation denoising, modify images toward piecewise constant functions. Though edge sharpness and location is well preserved, important information, encoded in image features like textures or small details, is often lost in the process. We suggest a simple way to better preserve textures, small details, or global information. This is done by adding a spatially varying fidelity term that controls the amount of denoising in any region of the image. This form is very simple, can be used for a variety of tasks in PDE-based image processing and computer vision, and is stable and meaningful from a mathematical point of view.

Signal and image enhancement is considered in the context of a new type of diffusion process that simultaneously enhances, sharpens, and denoises images. The nonlinear diffusion coefficient is locally adjusted according to image features such as edges, textures, and moments. As such, it can switch the diffusion process from a forward to a backward (inverse) mode according to a given set of criteria. This results in a forward-and-backward (FAB) adaptive diffusion process that enhances features while locally denoising smoother segments of the signal or image. The proposed method, using the FAB process, is applied in a super-resolution scheme. The FAB method is further generalized for color processing via the Beltrami flow, by adaptively modifying the structure tensor that controls the nonlinear diffusion process. The proposed structure tensor is neither positive definite nor negative, and switches between these states according to image features. This results in a forward-and-backward diffusion flow where different regions of the image are either forward or backward diffused according to the local geometry within a neighborhood.

Two nonlinear diffusion processes with time-dependent diffusion coefficients are presented. Both processes converge to nontrivial solutions, eliminating the need to impose an arbitrary diffusion stopping time, otherwise required in the implementation of most nonlinear diffusion processes. The two schemes employ nonlinear cooling mechanisms that preserve edges. One scheme is intended for general denoising, whereas the other is targeted for enhancement or segmentation of images. Simulation results indicate that the proposed schemes provide a stable and efficient tool for processing of noisy images.

Signal and image enhancement in the presence of noise is considered in the context of the scale-space approach. A modified dynamic process, based on the action of a nonlinear diffusion equation, is presented. The diffusion coefficient is adjusted according to the local gradient, intensity and other image properties, and as such also reverses its sign, i.e. switches from a forward to a backward (inverse) diffusion process according to a given criterion. This results in enhancement of transients and singularities in the one-dimensional case, and of edges in images, while locally denoising smoother segments of the signal or image. Regularization of the ill-posed inverse diffusion problem is discussed. Examples of both one-dimensional signals and images are presented.