Abstract. Existing research highlights the crucial role of topological priors in image segmentation, particularly in preserving essential structures, such as connectivity and genus. Accurately capturing these topological features often requires incorporating width-related information, including the thickness and length inherent to the image structures. However, traditional mathematical definitions of topological structures lack this dimensional width information, limiting methods like persistent homology from fully addressing practical segmentation needs. To overcome this limitation, we propose a novel mathematical framework that explicitly integrates width information into the characterization of topological structures. This method leverages persistent homology, complemented by smoothing concepts from PDEs, to modify local extrema of upper level sets. This approach enables the resulting topological structures to inherently capture width properties. We incorporate this enhanced topological description into variational image segmentation models. Using some proper loss functions, we are also able to design neural networks that can segment images with the required topological and width properties. Through variational constraints on the relevant topological energies, our approach successfully preserves essential topological invariants, such as connectivity and genus counts, simultaneously ensuring that segmented structures retain critical width attributes, including line thickness and length. Numerical experiments demonstrate the effectiveness of our method, showcasing its capability to maintain topological fidelity while explicitly embedding width characteristics into segmented image structures.

The elliptical shape prior information plays a vital role in improving the accuracy of image segmentation for specific tasks in medical and natural images. Existing deep learning-based segmentation methods, including the Segment Anything Model (SAM), often struggle to produce segmentation results with elliptical shapes efficiently. This paper proposes a new approach to integrate the prior of elliptical shapes into the deep learning-based SAM image segmentation techniques using variational methods. The proposed method establishes a parameterized elliptical contour field, which constrains the segmentation results to align with predefined elliptical contours. Utilizing the dual algorithm, the model seamlessly integrates image features with elliptical priors and spatial regularization priors, thereby greatly enhancing segmentation accuracy. By decomposing the SAM into four mathematical subproblems, we integrate the variational ellipse prior to design a new SAM network structure, ensuring that the segmentation output of the SAM consists of elliptical regions. Experimental results on some specific image datasets demonstrate an improvement over the original SAM. The codes are available in https://github.com/zhaoxinyum/SAM-ESP.

Normal reconstruction is crucial in non-line-of-sight (NLOS) imaging, as it provides key geometric and lighting information about hidden objects, which significantly improves reconstruction accuracy and scene understanding. However, jointly estimating normals and albedo expands the problem from matrix-valued functions to tensor-valued functions that substantially increasing complexity and computational difficulty. In this paper, we propose a novel joint albedo-surface reconstruction method, which utilizes the shape operator to control the variation rate of the normal field. It is the first attempt to apply regularization methods to the reconstruction of surface normals for hidden objects. By improving the accuracy of the normal field, it enhances detail representation and achieves high-precision reconstruction of hidden object geometry. The proposed method demonstrates robustness and effectiveness on both synthetic and experimental datasets. On transient data captured within 15 seconds, our surface normal-regularized reconstruction model produces more accurate surfaces than recently proposed methods and is 30 times faster than the existing surface reconstruction approach.

Multi-center star shape is a prevalent object shape feature, which has proven effective in model-based image segmentation methods. However, the shape field function induced by the multi-center star shape is non-smooth, and directly applying it to the data-driven image segmentation network architecture design may lead to instability in backpropagation. This paper proposes a convex combination star (CCS) shape, possessing multi-center star shape properties, and has the advantage of effectively controlling the shape of the region through a smooth field function. The sufficient condition of the proposed CCS shape can be combined into the image segmentation neural network structure design through the bridge between the variational segmentation model and the activation function of the data-driven method. Taking Segment Anything Model (SAM) and its improved version as backbone networks, we have shown that the segmentation network architecture with CCS shape properties can greatly improve the accuracy of segmentation results.

Graph neural networks have emerged as powerful tools for analyzing graph‐structured data, particularly in semisupervised node classification tasks. However, the conventional softmax classifier, widely used in such tasks, fails to leverage the spatial information inherent in graph structures. To address this limitation, we propose a graph similarity regularized softmax for graph neural networks, which incorporates nonlocal total variation regularization into the softmax function to explicitly capture graph structural information. The weights in the nonlocal gradient and divergence operators are determined based on the graph’s adjacency matrix. We implement this regularized softmax in two popular graph neural network architectures, GCN and GraphSAGE, and evaluate its performance on citation (assortative) and webpage linking (disassortative) datasets. Experimental results demonstrate that our method significantly improves node classification accuracy and generalization compared to baseline models. These findings highlight the effectiveness of the proposed regularized softmax in handling both assortative and disassortative graphs, offering a principled way to encode graph spatial information into graph neural network classifiers.

Non-line-of-sight (NLOS) imaging is an ill-posed problem to reconstruct hidden 3D scenes by leveraging photon time-of-flight information from diffusely reflected light. In the existing regularization models, the spatial residuals were handled by a single distribution, failing to account for the distinct characteristics of background and target objects. In this paper, we propose a novel NLOS reconstruction method that models the non-Gaussian residuals with a mixture distribution. Through a dual method, we derive an adaptive weighted residual model, where the weights generated in the dual space act as a zero-shot attention mechanism to control the contributions of different regions. The corresponding optimization problem can be effectively solved using the alternating minimization algorithm. Numerical experiments on both synthetic and real-world datasets demonstrate that our method surpasses the related existing approaches, achieving state-of-the-art performance. The code is available at:https://github.com/qiuxuanzhizi/AMD-NLOS-V1.

Morphological methods play a crucial role in remote sensing image processing, due to their ability to capture and preserve small structural details. However, most of the existing deep learning models for semantic segmentation are based on encoder-decoder architectures including U-Net and Segment Anything Model (SAM), where the downsampling process tends to discard fine details. In this paper, we propose a new approach that integrates learnable morphological skeleton prior into deep neural networks using the variational method. To address the difficulty in backpropagation in neural networks caused by the non-differentiability presented in classical morphological operations, we provide a smooth representation of the morphological skeleton and design a variational segmentation model integrating morphological skeleton prior by employing operator splitting and dual methods. Then, we integrate this model into the network architecture of SAM, which is achieved by adding a token to mask decoder and modifying the final sigmoid layer, ensuring the final segmentation results preserve the skeleton structure as much as possible. Experimental results on remote sensing datasets, including buildings, roads and water bodies, demonstrate that our method outperforms the original SAM on slender object segmentation and exhibits better generalization capability.

For effective image segmentation, it is crucial to employ constraints informed by prior knowledge about the characteristics of the areas to be segmented to yield favorable segmentation outcomes. However, the existing methods have primarily focused on priors of specific properties or shapes, lacking consideration of the general global shape similarity from a Contour Flow perspective. Furthermore, naturally integrating this contour flow prior image segmentation model into the activation functions of deep convolutional networks through mathematical methods is currently unexplored. In this paper, we establish a concept of global shape similarity based on the premise that two shapes exhibit comparable contours. Furthermore, we mathematically derive a contour flow constraint that ensures the preservation of global shape similarity. We propose two implementations to integrate the constraint with deep neural networks. Firstly, the constraint is converted to a shape loss, which can be seamlessly incorporated into the training phase for any learning-based segmentation framework. Secondly, we add the constraint into a variational segmentation model and derive its iterative schemes for solution. The scheme is then unrolled to get the architecture of the proposed CFSSnet. Validation experiments on diverse datasets are conducted on classic benchmark deep network segmentation models. The results indicate a great improvement in segmentation accuracy and shape similarity for the proposed shape loss, showcasing the general adaptability of the proposed loss term regardless of specific network architectures. CFSSnet shows robustness in segmenting noise-contaminated images, and inherent capability to preserve global shape similarity.

In this article, we explore the Mechanism Design aspects of the Maximum Vertex-Weighted \(b\) -matching (MVbM) problem on bipartite graphs \((A\cup T,E)\) . The set \(A\) comprises agents, while \(T\) represents tasks. The set \(E\) , which connects \(A\) and \(T\) , is the private information of either agents or tasks. In this framework, we investigate three mechanisms— \(\mathbb{M}_{BFS}\) , \(\mathbb{M}_{DFS}\) , and \(\mathbb{M}_{G}\) . We examine scenarios in which either agents or tasks are strategic and report their adjacent edges to one of the three mechanisms. In both cases, we assume that the strategic entities are bounded by their statements: They can hide edges, but they cannot report edges that do not exist. First, we consider the case in which agents can manipulate. In this framework, \(\mathbb{M}_{BFS}\) and \(\mathbb{M}_{DFS}\) are optimal but not truthful. By characterizing the Nash Equilibria induced by \(\mathbb{M}_{BFS}\) and \(\mathbb{M}_{DFS}\) , we reveal that both mechanisms have a Price of Anarchy ( \(PoA\) ) and Price of Stability ( \(PoS\) ) of \(2\) . These efficiency guarantees are tight; no deterministic mechanism can achieve a lower \(PoA\) or \(PoS\) . In contrast, the third mechanism, \(\mathbb{M}_{G}\) , is not optimal, but truthful and its approximation ratio is \(2\) . We demonstrate that this ratio is optimal; no deterministic and truthful mechanism can outperform it. We then shift our focus to scenarios where tasks can exhibit strategic behavior. In this case, \(\mathbb{M}_{BFS}\) , \(\mathbb{M}_{DFS}\) , and \(\mathbb{M}_{G}\) all maintain truthfulness, making \(\mathbb{M}_{BFS}\) and \(\mathbb{M}_{DFS}\) truthful and optimal mechanisms. In conclusion, we investigate the manipulability of \(\mathbb{M}_{BFS}\) and \(\mathbb{M}_{DFS}\) through experiments on randomly generated graphs. We observe that (i) \(\mathbb{M}_{BFS}\) is less prone to be manipulated by the first agent than \(\mathbb{M}_{DFS}\) , and (ii) \(\mathbb{M}_{BFS}\) is more manipulable on instances in which the total capacity of the agents is equal to the number of tasks. 1

Abstract. The classical Mumford–Shah (MS) model has been successful in some medical image segmentation tasks, providing segmentation results with smooth boundaries of objects. However, the MS model, which operates at the pixel level of the images, faces challenges when dealing with medical images with low contrast or unclear edges. In this paper, we begin by using a feature extractor to capture high-dimensional deep features that contain more comprehensive semantic information than pixel-level data alone. Inspired by the MS model, we develop a variational model that incorporates threshold dynamics (TD) regularization for segmenting each feature. We obtain the final segmentation result for the original image by assembling segmentation results of all the features. This process results in MS-MGNet, a lightweight trainable segmentation network with a similar architecture to many encoder–decoder networks. The intermediate layers of MS-MGNet are designed by unrolling the numerical scheme based on the multigrid method for solving the variational model. We provide interpretability for the encoder–decoder architecture by elucidating the roles of each layer and offering explanations of the underlying mathematical models. By incorporating the TD regularizer, we integrate spatial priors from the variational models into the network architecture, resulting in better segmentation results with smoother edges and a certain robustness to noise. Compared to some relevant methods, experimental results on the selected data sets with low contrast or unclear edges show that the proposed method can achieve better segmentation performance with fewer parameters, even when trained on smaller data sets.

Abstract. High-dimensional deep features extracted by convolutional neural networks have nonlocal self-similarity. However, incorporating this nonlocal prior of deep features into deep network architectures with an interpretable variational framework is rarely explored. In this paper, we propose a learnable nonlocal self-similarity deep feature network for image denoising. Our method is motivated by the fact that the high-dimensional deep features obey a mixture probability distribution based on the Parzen–Rosenblatt window method. Then a regularizer with learnable nonlocal weights is proposed by considering the dual representation of the log-probability prior of the deep features. Specifically, the nonlocal weights are introduced as dual variables that can be learned by unrolling the associated numerical scheme. This leads to nonlocal modules (NLMs) in newly designed networks. Our method provides a statistical and variational interpretation for the nonlocal self-attention mechanism widely used in various networks. By adopting nonoverlapping window and region decomposition techniques, we can significantly reduce the computational complexity of nonlocal self-similarity, thus enabling parallel computation of the NLM. The solution to the proposed variational problem can be formulated as a learnable nonlocal self-similarity network for image denoising. This work offers a novel approach for constructing network structures that consider self-similarity and nonlocality. The improvements achieved by this method are predictable and partially controllable. Compared with several closely related denoising methods, the experimental results show the effectiveness of the proposed method in image denoising.

The multireference alignment (MRA) problem is to estimate an underlying signal from a large number of noisy circularly-shifted observations. The existing methods are under the hypothesis of a single Gaussian noise. However, the hypothesis of a single-type noise is inefficient for solving practical problems like single particle cryo-EM. In this paper, we derive an adaptive variational model by combining maximum a posteriori (MAP) estimation and the soft-max method under the assumption of Gaussian mixture noise. There are two adaptive weights for detecting cyclical shifts and types of noise separately. The existence of a minimizer is mathematically proved. We design a novel algorithm for the proposed model using the alternating direction iterative method and the augmented Lagrange method. There are some convergence analyses for the proposed algorithm under some conditions. The numerical results show that the proposed model performs better than the existing methods in that the level of one Gaussian noise is high and the other is low.

Image segmentation is a crucial but challenging task that has many applications. In medical imaging, for instance, intensity inhomogeneity and noise are common. In thigh muscle images, different muscles are closely packed together and there are often no clear boundaries between them. Intensity based segmentation models cannot separate one muscle from another. To solve such problems, in this work we present a segmentation model with adaptive spatial priors from joint registration. This model combines segmentation and registration in a unified framework to leverage their positive mutual influence. The segmentation is based on a modified Gaussian mixture model, which integrates intensity inhomogeneity and spatial smoothness. The registration plays the role of providing a shape prior. We adopt a modified sum of squared difference fidelity term and Tikhonov regularity term for registration and also utilize a Gaussian pyramid and parametric method for robustness. The connection between segmentation and registration is guaranteed by the cross entropy metric that aims to make the segmentation map (from segmentation) and deformed atlas (from registration) as similar as possible. This joint framework is implemented within a constraint optimization framework, which leads to an efficient algorithm. We evaluate our proposed model on synthetic and thigh muscle MR images. Numerical results show the improvement as compared to segmentation and registration performed separately and other joint models.

Non-Gaussian residual error and noise are common in the real applications, and they can be efficiently addressed by some non-quadratic fidelity terms in the classic variational method. However, they have not been well integrated into the architectures design in the convolutional neural networks (CNN) based image denoising method. In this paper, we propose a deep learning approach to handle non-Gaussian residual error. Our method is developed on an universal approximation property for the probability density functions of the non-Gaussian error/noise. By considering the duality of the maximum likelihood estimation for the non-Gaussian error, an adaptive weighting strategy can be derived for image fidelity. To get a good image prior, a learnable regularizer is adopted. Solving such a problem iteratively can be unrolled as a weighted residual CNN architecture. The main advantage of our method is that the weighted residual block can well handle the non-Gaussian residual, especially for the noise with non-uniformly spatial distribution. Numerical results show that it has better performance on non-Gaussian noise (e.g. Gaussian mixture, random-valued impulse noise) removal than the related existing methods.

Image segmentation is a fundamental research topic in image processing and computer vision. In recent decades, researchers developed a large number of segmentation algorithms for various applications. Among these algorithms, the normalized cut (Ncut) segmentation method is widely applied due to its good performance. The Ncut segmentation model is an optimization problem whose energy is defined on a specifically designed graph. Thus, the segmentation results of the existing Ncut method are largely dependent on a preconstructed similarity measure on the graph since this measure is usually given empirically by users. This flaw will lead to some undesirable segmentation results. In this paper, we propose an Ncut-based segmentation algorithm by integrating an adaptive similarity measure and spatial regularization. The proposed model combines the Parzen--Rosenblatt window method, nonlocal weights entropy, Ncut energy, and regularizer of phase field in a variational framework. Our method can adaptively update the similarity measure function by estimating some parameters. This adaptive procedure enables the proposed algorithm to find a better similarity measure for classification than the Ncut method. We provide some mathematical interpretation of the proposed adaptive similarity from multiple viewpoints, such as statistics and convex optimization. In addition, the regularizer of phase field can guarantee that the proposed algorithm has a robust performance in the presence of noise, and it can also rectify the similarity measure with a spatial priori. The well-posed theory such as the existence of the minimizer for the proposed model is given in the paper. Compared with some existing segmentation methods such as the traditional Ncut-based model and the classical Chan--Vese model, the numerical experiments show that our method can provide promising segmentation results.

Hyperspectral image (HSI) super-resolution refers to enhancing the spatial resolution of a 3-D image with many spectral bands (slices). It is a seriously ill-posed problem when the low-resolution (LR) HSI is the only input. It is better solved by fusing the LR HSI with a high-resolution (HR) multispectral image (MSI) for a 3-D image with both high spectral and spatial resolution. In this article, we propose a novel nonnegative and nonlocal 4-D tensor dictionary learning-based HSI super-resolution model using group-block sparsity. By grouping similar 3-D image cubes into clusters and then conduct super-resolution cluster by cluster using 4-D tensor structure, we not only preserve the structure but also achieve sparsity within the cluster due to the collection of similar cubes. We use 4-D tensor Tucker decomposition and impose nonnegative constraints on the dictionaries and group-block sparsity. Numerous experiments demonstrate that the proposed model outperforms many state-of-the-art HSI super-resolution methods.

In this paper, the traditional model based variational methods and deep learning based algorithms are naturally integrated to address mixed noise removal, specially for Gaussian mixture noise and Gaussian-impulse noise removal problem. To be different from single type noise (e.g. Gaussian) removal, it is a challenge problem to accurately discriminate noise types and levels for each pixel. We propose a variational method to iteratively estimate the noise parameters, and then the algorithm can automatically classify the noise according to the different statistical parameters. The proposed variational problem can be separated into regularization, synthesis, parameters estimation and noise classification four steps with the operator splitting scheme. Each step is related to an optimization subproblem. To enforce the regularization, the deep learning method is employed to learn the natural images prior. Compared with some model based regularizations, the CNN regularizer can significantly improve the quality of the restored images. Compared with some learning based methods, the synthesis step can produce better reconstructions by analyzing the types and levels of the recognized noise. In our method, the convolution neutral network (CNN) can be regarded as an operator which associated to a variational functional. From this viewpoint, the proposed method can be extended to many image reconstruction and inverse problems. Numerical experiments in the paper show that our method can achieve some state-of-the-art results for Gaussian mixture noise and Gaussian-impulse noise removal.

In this paper, we propose an image segmentation model that incorporates convexity shape priori using level set representations. In the past decade, several discrete and continuous methods have been developed to solve this problem. Our method comes from the observation that the signed distance function of a convex region must be a convex function. Based on this observation, we transfer the complicated geometrical convexity shape priori into some simple constraints on the signed distance function. We propose a simple algorithm to keep these constraints exactly. The proposed method could be easily applied to level set based segmentation models, such as the well-known Chan-Vese mode and the active contour models. By setting some good initial curves, the proposed method can easily segment convex objects from images with complicated background. We demonstrate the performance of the proposed methods on both synthetic images and real images, as well as the comparison to some state-of-the-art methods.

To achieve high scene classification performance of high spatial resolution remote sensing images (HSR-RSIs), it is important to learn a discriminative space in which the distance metric can precisely measure both similarity and dissimilarity of features and labels between images. While the traditional metric learning methods focus on preserving interclass separability, label consistency (LC) is less involved, and this might degrade scene images classification accuracy. Aiming at considering intraclass compactness in HSR-RSIs, we propose a discriminative distance metric learning method with LC (DDML-LC). The DDML-LC starts from the dense scale invariant feature transformation features extracted from HSR-RSIs, and then uses spatial pyramid maximum pooling with sparse coding to encode the features. In the learning process, the intraclass compactness and interclass separability are enforced while the global and local LC after the feature transformation is constrained, leading to a joint optimization of feature manifold, distance metric, and label distribution. The learned metric space can scale to discriminate out-of-sample HSR-RSIs that do not appear in the metric learning process. Experimental results on three data sets demonstrate the superior performance of the DDML-LC over state-of-the-art techniques in HSR-RSI classification.

This paper proposes a general weighted l(2)-l(0) norms energy minimization model to remove mixed noise such as Gaussian-Gaussian mixture, impulse noise, and Gaussian-impulse noise from the images. The approach is built upon maximum likelihood estimation framework and sparse representations over a trained dictionary. Rather than optimizing the likelihood functional derived from a mixture distribution, we present a new weighting data fidelity function, which has the same minimizer as the original likelihood functional but is much easier to optimize. The weighting function in the model can be determined by the algorithm itself, and it plays a role of noise detection in terms of the different estimated noise parameters. By incorporating the sparse regularization of small image patches, the proposed method can efficiently remove a variety of mixed or single noise while preserving the image textures well. In addition, a modified K-SVD algorithm is designed to address the weighted rank-one approximation. The experimental results demonstrate its better performance compared with some existing methods.