This work presents a progressive image vectorization technique that reconstructs the raster image as layer-wise vectors from semantic-aligned macro structures to finer details. Our approach introduces a new image simplification method leveraging the feature-average effect in the Score Distillation Sampling mechanism, achieving effective visual abstraction from the detailed to coarse. Guided by the sequence of progressive simplified images, we propose a two-stage vectorization process of structural buildup and visual refinement, constructing the vectors in an organized and manageable manner. The resulting vectors are layered and well-aligned with the target image’s explicit and implicit semantic structures. Our method demonstrates high performance across a wide range of images. Comparative analysis with existing vectorization methods highlights our technique’s superiority in creating vectors with high visual fidelity, and more importantly, achieving higher semantic alignment and more compact layered representation.

Current blind image quality assessment (BIQA) models usually lack adaptability to the test data with distribution shifts to the training data. This inspires an investigation into test time adaptation (TTA) methods to address distribution shifts between training and test data. However, existing methods mainly focus on simple feature alignment strategies, which may lead to incorrect knowledge generalization. To this issue, we propose a prior-guided test time adaptation (PGTA-IQA) for blind image quality assessment. Concretely, we extract the quality prior knowledge from the pre-trained BIQA model through clustering. The extracted quality prior knowledge forms the foundation for subsequent optimizations. These optimizations are carried out from two complementary perspectives: inter-cluster and intra-cluster. From the inter-cluster perspective, we propose a confident rank learning approach which consists of a relative quality matrix (RQM) and a confidence filtering strategy (CFS) to generate the high-confident quality rankings. From the intra-cluster perspective, we propose a selective feature alignment approach by only aligning the closest neighboring samples within the same cluster to reduce the impact of noisy labels. The experimental results demonstrate the effectiveness of the proposed approaches.

Deep-sea images suffer from extreme light attenuation and non-uniform illumination caused by artificial light sources. To get rid of limitation aroused by low-quality training data, we propose an unsupervised approach for deep-sea image enhancement based on the prior-to-image framework, termed as OptiDiff. Instead of learning mapping from paired underwater dataset, the framework for OptiDiff is trained on air image dataset. Specifically, four optical-invariant priors (OIPs) are used for guiding the stable diffusion model to recover degraded underwater image. One of the utilized OIPs is particularly designed for recover blurred details in background. Besides, to simulate the domain shift between air and underwater images, a channel attenuation strategy derived from characteristics of real-world underwater image is equipped with the framework. Extensive experimental results demonstrate the superiority of the proposed OptiDiff on restoring image from low contrast, low visibility, and severe blur, also showing robustness on shallow-water datasets. Code is available at https://github.com/Miaaaaaa1024/OptiDiff.

Visual foundation models (VFMs) have demonstrated impressive generalization capabilities in computer vision tasks. Previous studies show that fine-tuning VFMs with learnable tokens can achieve better generalization performance than full-parameter fine-tuning. The problem we need to address is how to learn the tokens that focus on the content information while ignoring the influence of style. For this purpose, we propose a novel Dual-Branch Content-enhanced Token (DBCT) learning framework. Specifically, we construct a style-suppressing branch, which contains a Style-sensitive Channel Suppression (SCS) module to transform the frozen VFM features into style-suppressed features, enabling the learning of style-invariant tokens. In addition, to compensate for the content degradation caused by the style-suppressing branch, we introduce a content-preserving branch that directly takes the frozen VFM features as input to learn content-focused tokens. Meanwhile, we propose a Token-query Linking (TLink) strategy to connect the two sets of tokens with the queries in the decoder. Through extensive experiments, our method achieves advanced results on various benchmarks.

The total variation models are popular in various image processing tasks such as smoothing, decomposition and depth estimation. However, solving these models are challenging. They can be solved by traditional iterative algorithms that require a large number of iterations to converge or deep neural networks that have a large number of trainable parameters. In this paper, we propose a novel Anderson accelerated residual solver (AARS) for these models. Although our method is iterative, it requires much less iteration numbers than the traditional iterative methods, thanks to the Anderson acceleration. Mean-while, it can theoretically guarantee to converge to the global optimal solution. This is theoretically proved and numerically confirmed. Several numerical experiments are conducted to show the effectiveness and efficiency of the proposed solver. It can be applied in various image processing tasks where solving the total variation models is necessary.

Edge-preserving image filters are fundamentally important for various computer vision tasks. Among these filters, one-sided box filter (OSBF) is designed to achieve edge preserving and high computation performance. To further improve its computation performance, in this paper, we propose to accelerate the one-sided box filter by Field Programmable Gate Arrays (FPGA). We use the ZYNQ7100 board with ARM Cortex-A9 cores. And the accelerated version is 4 times faster than the original one-sided box filter. We further compared it with other classical filters such as box filter, median filter, bilateral filter and guided filter. Several numerical experiments confirm that the FPGA accelerated OSBF outperforms other edge-preserving methods in terms of edge-preserving and computation performance. Thanks to its high performance, the proposed method can be applied in a large range of applications such as smart phones, satellite imaging, autonomous driving and smart microscopes, especially for the real time applications.

Thermal infrared images (TIIs) can be distorted by multiple factors, resulting in noise, low contrast, limited dynamic range, and fuzziness, which greatly impede their usefulness. It is crucial to evaluate the quality of TIIs. Unfortunately, there have been very few attempts to study this problem. In this study, we collected 1,000 authentically distorted TIIs using thermal infrared acquisition equipment and conducted strict subjective experiments to obtain a thermal infrared image quality assessment (IQA) database. Each image’s quality score was obtained under strict scoring rules. Finally, we investigated the feasibility of several no-reference (NR) IQA methods in quality assessment of TIIs. We found that existing NR-IQA methods achieve ordinary performance in such a task, and there is an urgent need to develop a specific IQA methods for TIIs. The findings together with the constructed database are expected to pave the way for the development of more advanced IQA methods for further development of this field.

Although there are many iterative solvers for total variation models, few attention has been paid on the fast and effective approximation to their optimal solutions. In this paper, we propose a closed-form filter that can efficiently and effectively approximate the optimal solution of total variation models. This filter has linear computation complexity $O(n)$ with respect to the total number of pixels and constant computation complexity $O(1)$ with respect to the window radius. Taking such filter as an initialization, our method can significantly accelerate all previous iterative solvers. Numerical experiments confirms that our initialization is roughly equivalent to $\mathbf{5 0}$ iterations in the iterative method but $\mathbf{1 0} \times$ faster. The proposed method can be applied in all total variation models to accelerate the optimization process, such as image smoothing, image reconstruction and optical flow estimation.

Abstract Mesh denoising is a fundamental yet challenging task. Most of the existing data‐driven methods only consider the zero‐order information (vertex location) and first‐order information (face normal). However, higher‐order geometric information (such as curvature) is more descriptive for the shape of the mesh. Therefore, in order to impose such high‐order information, this paper proposes a novel Curvature‐Driven Multi‐Stream Graph Convolutional Neural Network (CDMS‐Net) architecture. CDMS‐Net has three streams, including curvature stream, face normal stream and vertex stream, where the curvature stream focuses on the high‐order Gaussian curvature information. Moreover, CDMS‐Net proposes a novel block based on residual dense connections, which is used as the core component to extract geometric features from meshes. This innovative design improves the performance of feature‐preserving denoising. The plug‐and‐play modular design makes CDMS‐Net easy to be implemented. Multiple sets of ablation study are carried out to verify the rationality of the CDMS‐Net. Our method establishes new state‐of‐the‐art mesh denoising results on publicly available datasets.

This paper proposes a multiscale residual solver for total variation regularized models. The proposed solver has three important properties. First, the proposed algorithm is theoretically guaranteed to converge. Second, the proposed method can numerically reach the same global optimal solution as the classical methods. This fact is confirmed on image datasets. Third, the proposed solver is faster than the classical methods. For high resolution images, our solver can be three orders of magnitude faster. The proposed method can be adopted in the applications where total variation regularization is imposed, such as denoising, smoothing, optical flow, and inpainting.

Guided filter is a popular filter thanks to its effectiveness in edge preserving and computation efficiency. On the other hand, total variation is a popular prior thanks to its mathematical properties. In this paper, we impose the total variation prior into guided filter, leading to a novel Total Variation Guided Filter (TVGF) that has both advantages from guided filter and total variation prior. First, TVGF has a closed-form expression and also a linear computation complexity. Our experiment confirms that TVGF is 1300+ times faster than traditional iterative solvers for total variation models. Second, thanks to the total variation, TVGF preserves edges better than the original guided filter and leads to much less artifacts. TVGF is also two times faster than the guided filter. We theoretically derive the TVGF and numerically confirm its effectiveness and efficiency. TVGF can be used in a large range of edge-aware applications, such as image smoothing, dehazing, depth estimation and optical flow estimation.

3D meshes are getting popular in both research and industry. However, the meshes obtained via the 3D scanning equipment frequently contain a high level of noise. In this paper, we present a Gaussian Curvature Driven Multi-stream Network (GCM-Net) based on graph convolutional networks. This network can remove the noise while preserving the essential features during the 3D mesh denoising process. Our method is the first attempt to apply the high-order feature (i.e., Gaussian curvature) in the denoising task, which is more descriptive for the shape of the mesh. GCM-Net consists of curvature stream, vertex stream, and face normal stream, where the curvature stream focuses on the high-order Gaussian curvature feature of 3D mesh. Our method achieves state-of-the-art results on a publicly available dataset, demonstrating its effectiveness. The proposed method can be applied in various applications, such as 3D human body modeling, metaverse, object tracking and biomedical visualization.

Given a new 6DoF camera pose in an indoor environment, we study the challenging problem of predicting the view from that pose based on a set of reference RGBD views. Existing explicit or implicit 3D geometry construction methods are computationally expensive while those based on learning have predominantly focused on isolated views of object categories with regular geometric structure. Differing from the traditional render-inpaint approach to new view synthesis in the real indoor environment, we propose a conditional generative adversarial neural network (P2I-NET) to directly predict the new view from the given pose. P2I-NET learns the conditional distribution of the images of the environment for establishing the correspondence between the camera pose and its view of the environment, and achieves this through a number of innovative designs in its architecture and training lost function. Two auxiliary discriminator constraints are introduced for enforcing the consistency between the pose of the generated image and that of the corresponding real world image in both the latent feature space and the real world pose space. Additionally a deep convolutional neural network (CNN) is introduced to further reinforce this consistency in the pixel space. We have performed extensive new view synthesis experiments on real indoor datasets. Results show that P2I-NET has superior performance against a number of NeRF based strong baseline models. In particular, we show that P2I-NET is 40 to 100 times faster than these competitor techniques while synthesising similar quality images. Furthermore, we contribute a new publicly available indoor environment dataset containing 22 high resolution RGBD videos where each frame also has accurate camera pose parameters.

Traditional computing methods for curvatures require the image to be second-order differentiable. Such requirement is not always satisfied, especially at sharp edges. In this paper, we propose a novel method that can compute curvatures without requiring the image second-order differentiable. We first establish the link between various curvatures and the standard Laplace operator. Then we propose to compute curvatures by a half kernel Laplace method. Our method has a smaller support region and thus is more accurate than traditional methods. It can be further adopted to compute curvature, mean curvature, and weighted mean curvature. Our method is compared with the classical schemes on both synthetic and real images, showing its effectiveness and efficiency.

With the rapid development of weak-illumination imaging technology, low-light images have brought new challenges to quality of experience and service. However, developing a robust quality indicator for authentic low-light distortions in-the-wild remains a major challenge in practical quality control systems. In this paper, we develop a new super-complete comparative representation (S-CCR) for the region-level quality inference of low-light images. Specifically, we excavate the color, luminance, and detail quality evidence for the feature embedding guidance of comparative representation based on the human visual characteristics. Moreover, we decompose the inputs into a super-complete feature group so that the image quality of each region can be fully represented, which allows to preserve the distinctiveness, distinguishability, and consistency. Finally, we further establish a comparative domain alignment method, so that the comparative representation of an unseen image can be aligned with respect to the quality features of already-seen ones. Extensive experiments on the benchmark dataset validate the superiority of our S-CCR over 11 competing methods on authentic distortions.

In this paper, we propose a novel algorithm for 3D mesh filtering (Structural Adaptive Filtering, SAF) based on mesh structural adaptation. As we all know, 3D meshes mainly have three types of geometric features: corners, edges, and planes. Therefore, we designed a protection mechanism for these three types of features to achieve the feature-preserving denoising. In the first step, for the faces normals to be processed, we build a variable set of similarity between the face normal and the neighborhood faces normal, calculate their coefficient of variation, variance, and quartile difference, and then select the neighborhood face normals with high similarity to update the current normals through self adaptation of these variables. In the second step, all vertices complete the iterative update of vertex coordinates according to the filtered face normals. Unlike existing 3D mesh denoising algorithms, which have too many parameters to manually set thresholds and are sensitive to parameters, SAF is based on the geometry of local faces (no need to manually set denoising thresholds). SAF only needs to set the iterative parameters to complete high-performance feature-preserving filtering, which has high practical value. We demonstrate through extensive experimental data that SAF outperforms or is comparable to state-of-the-art methods in feature-preserving denoising at different noise levels.

In conditions such as inclement weather or in-sufficient lighting at night, video captured by camera equipment may be insufficiently bright and have low contrast. With the popularity of ultra-high-definition video images, its ultra-high resolution makes the performance requirements of video processing algorithms more and more stringent. Aiming at these problems, this paper proposes a curvature-based real-time brightness adjustment algorithm for ultra-high-definition video. The algorithm is implemented in two steps: global brightness adaptive enhancement and local contrast adaptive enhancement. The experimental results show that the algorithm proposed in this paper not only highlights the details of low-brightness areas of video images, but also avoids excessive enhancement of high-brightness areas. It is superior or comparable to the comparison algorithm in both subjective visual effects and objective evaluation indicators. In addition, because the algorithm in this paper is simple and easy to implement, it can be easily implemented in parallel, and can be applied to real-time processing of ultra-high-definition video, which has high practical value.

Weighted Gaussian curvature is an important smoothness measurement for images. However, its conventional computation scheme has low performance, low accuracy and requires that the input image must be second order differentiable. To tackle these three issues, we propose a novel discrete computation scheme for the weighted Gaussian curvature. Our scheme does not require the second order differentiability. Moreover, our scheme is more accurate, has smaller support region and computationally more efficient than the conventional schemes. Therefore, our scheme holds promise for a large range of applications where the weighted Gaussian curvature is needed, for example, image smoothing, cartoon texture decomposition, optical flow estimation, etc.

This paper presents a quarter Laplacian filter that can preserve corners and edges during image smoothing. Its support region is $2\times 2$, which is smaller than the $3\times 3$ support region of the classical Laplacian filter. Thus, it is more local. Moreover, this filter can be implemented via the classical box filter, leading to high performance for real time applications. Finally, we show its edge preserving property in several image processing tasks, including image smoothing, texture enhancement, and low-light image enhancement. The proposed filter can be adopted in a wide range of image processing applications.

Local windows are routinely used in computer vision and almost without exception the center of the window is aligned with the pixels being processed. We show that this conventional wisdom is not universally applicable. When a pixel is on an edge, placing the center of the window on the pixel is one of the fundamental reasons that cause many filtering algorithms to blur the edges. Based on this insight, we propose a new Side Window Filtering (SWF) technique which aligns the window's side or corner with the pixel being processed. The SWF technique is surprisingly simple yet theoretically rooted and very effective in practice. We show that many traditional linear and nonlinear filters can be easily implemented under the SWF framework. Extensive analysis and experiments show that implementing the SWF principle can significantly improve their edge preserving capabilities and achieve state of the art performances in applications such as image smoothing, denoising, enhancement, structure-preserving texture-removing, mutual-structure extraction, and HDR tone mapping. In addition to image filtering, we further show that the SWF principle can be extended to other applications involving the use of a local window. Using colorization by optimization as an example, we demonstrate that implementing the SWF principle can effectively prevent artifacts such as color leakage associated with the conventional implementation. Given the ubiquity of window based operations in computer vision, the new SWF technique is likely to benefit many more applications.

Soft tissue in X-ray images obscures the bone structure such that the details on bones are not clear. Conventional methods simultaneously enhance the image contrast for the soft tissue and the bones. Here we propose to remove all soft tissue in X-ray images, making the bone structure clear. For this purpose, we first propose a half window dark channel prior and a half window guided filter. Then, we apply this prior and filter on X-ray images. After processing, the bone details in X-ray images become clear and sharp. Several experiments confirm the effectiveness and efficiency of our method. Our method can be used for bone segmentation, classification, recognition, and clinical diagnosis.

Ill-posed problems are very common in many image processing and computer vision tasks. To solve such problems, a regularization must be imposed. In this paper, we argue why mean curvature is a good regularization for these tasks. From a geometry point of view, we show that minimizing mean curvature is to assume that the ground truth is a piece-wise minimal surface. From a statistics point of view, we show that the mean curvature from natural images is sparse. From an optimization point of view, we show that the gradient of mean curvature regularization can be numerically approximated by very simple filters. This fact significantly simplifies the optimization procedure. Thus, the link between these filters and the gradient of mean curvature regularization is established in this paper. In summary, mean curvature is a proper regularization for various ill-posed problems in image processing and computer vision.

One of the methods for stratifying different molecular classes of breast cancer is the Nottingham prognostic index plus, which uses breast cancer relevant biomarkers to stain tumor tissues prepared on tissue microarray (TMA). To determine the molecular class of the tumor, pathologists will have to manually mark the nuclei activity biomarkers through a microscope and use a semi-quantitative assessment method to assign a histochemical score (H-Score) to each TMA core. Manually marking positively stained nuclei is a time-consuming, imprecise, and subjective process, which will lead to inter-observer and intra-observer discrepancies. In this paper, we present an end-to-end deep learning system, which directly predicts the H-Score automatically. Our system imitates the pathologists' decision process and uses one fully convolutional network (FCN) to extract all nuclei region (tumor and non-tumor), a second FCN to extract tumor nuclei region, and a multi-column convolutional neural network, which takes the outputs of the first two FCNs and the stain intensity description image as an input and acts as the high-level decision making mechanism to directly output the H-Score of the input TMA image. To the best of our knowledge, this is the first end-to-end system that takes a TMA image as the input and directly outputs a clinical score. We will present experimental results, which demonstrate that the H-Scores predicted by our model have very high and statistically significant correlation with experienced pathologists' scores and that the H-Score discrepancy between our algorithm and the pathologists is on par with the inter-subject discrepancy between the pathologists.

Box filter is a fundamental filter in image processing. However, it can not preserve edges or corners. In this paper, we present a simple but novel method that can make box filter both edge and corner preserving. More specifically, we combine the box filter with sub-window regression to achieve this task. This filter inherits some properties from box filter, such as O(1) running time with respect to the window radius. After analyzing its parameters, we show its corner and edge preserving property on real images and compare it with Guided filter.

We take advantage of the popularity of deep convolutional neural networks (CNNs) and have developed a very simple image quality assessment method that rivals state of the art. We show that convolutional layer outputs (deep features) of a CNN compute the local structural information of spatial regions of different sizes in the input image. The learned convolutional kernels contain a much richer set of weights thus capturing much more local structural information than hand crafted ones. As the deep features learned from large datasets already contain very rich multi-resolutional structural image information, they can be directly used to calculate visual distortion of an image and it is not necessary to introduce further complicated computational process. We will present experimental results to demonstrate that this is indeed the case, and that simple cosine distance of the deep features is as good as state the art methods for full reference image quality assessment.

Mean curvature has been shown a good regularization for many image processing tasks. Computing mean curvature, however, usually requires the image at least twice differentiable, which is an issue for discrete images, especially at edges. In this paper, we present several linear schemes to approximate the mean curvature of discrete images, based on Euler Theorem from differential geometry. We further compare these schemes with the traditional formula in terms of accuracy, computational efficiency, convexity, etc. The experiments confirm that these schemes are good approximations to the mean curvature of discrete images.

In image processing, the rapid approximate solution of variational problems involving generic data-fitting terms is often of practical relevance, for example in real-time applications. Variational solvers based on diffusion schemes or the Euler-Lagrange equations are too slow and restricted in the types of data-fitting terms. Here, we present a filter-based approach to reduce variational energies that contain generic data-fitting terms, but are restricted to specific regularizations. Our approach is based on reducing the regularization part of the variational energy, while guaranteeing non-increasing total energy. This is applicable to regularization-dominated models, where the data-fitting energy initially increases, while the regularization energy initially decreases. We present fast discrete filters for regularizers based on Gaussian curvature, mean curvature, and total variation. These pixel-local filters can be used to rapidly reduce the energy of the full model. We prove the convergence of the resulting iterative scheme in a greedy sense, and we show several experiments to demonstrate applications in image-processing problems involving regularization-dominated variational models.

The mean curvature has been shown a proper regularization in various ill-posed inverse problems in signal processing. Traditional solvers are based on either gradient descent methods or Euler Lagrange Equation. However, it is not clear if this mean curvature regularization term itself is convex or not. In this paper, we first prove that the mean curvature regularization is convex if the dimension of imaging domain is not larger than seven. With this convexity, all optimization methods lead to the same global optimal solution. Based on this convexity and Bernstein theorem, we propose an efficient filter solver, which can implicitly minimize the mean curvature. Our experiments show that this filter is at least two orders of magnitude faster than traditional solvers.

We present a variational model with local weighted Gaussian curvature as regularizer. We show its convexity for an area-weight function and provide a closed-form solution for this case. The corresponding regularization coefficient has a theoretical bound. Moreover, we prove that the model is convex for a wide range of weight functions and show that it can be efficiently solved using splitting techniques. Finally, we demonstrate several applications of the model in image de-noising, smoothing, texture decomposition, image sharpening, and regularization-coefficient optimization.