Ultra-High-Definition (UHD) image restoration has acquired remarkable attention due to its practical demand. In this paper, we construct UHD snow and rain benchmarks, named UHD-Snow and UHD-Rain, to remedy the deficiency in this field. The UHD-Snow/UHD-Rain is established by simulating the physics process of rain/snow into consideration and each benchmark contains 3200 degraded/clear image pairs of 4K resolution. Furthermore, we propose an effective UHD image restoration solution by considering gradient and normal priors in model design, thanks to these priors’ spatial and detail contributions. Specifically, our method contains two branches: (a) feature fusion and reconstruction branch in high-resolution space and (b) prior feature interaction branch in low-resolution space. The former learns high-resolution features and fuses prior-guided low-resolution features to reconstruct clear images, while the latter utilizes normal and gradient priors to mine useful spatial features and detail features to guide high-resolution recovery better. To better utilize these priors, we introduce single prior feature interaction and dual prior feature interaction, where the former respectively fuses normal and gradient priors with high-resolution features to enhance prior ones, while the latter calculates the similarity between enhanced prior ones and further exploits dual guided filtering to boost the feature interaction of dual priors. We conduct experiments on both new and existing public datasets and demonstrate the state-of-the-art performance of our method on UHD image low-light enhancement, dehazing, deblurring, desnowing, and deraining. The source codes and benchmarks are available at https://github.com/wlydlut/UHDDIP.

This paper proposes a starting point controlled (SPC) diffusion model for general-purpose image restoration, which directly models the image degradation process as a diffusion process. The key idea of the SPC approach is to transform a high-quality image into a degraded counterpart, which is a low-quality image with added Gaussian noise as the starting point for the reverse process rather than pure Gaussian noise. Then, by simulating the corresponding SPC reverse-time process, we can restore the original high-quality image from its low-quality image without requiring any task-specific prior knowledge. Crucially, the IR-SPC method has a closed-form solution, allowing us to compute the true posterior probability and learn it with a neural network. Furthermore, we introduce a maximum likelihood objective to optimize the reverse trajectory that stabilizes the training and improves the restoration results. Additionally, we also propose optimized training and sampling algorithms to improve the computational efficiency and algorithmic consistency of the IR-SPC framework. Extensive experimental results show that our proposed method achieves highly competitive performance in quantitative comparisons of perceptual metrics, setting a new state-of-the-art perception metrics on restoration tasks.

Multi-modal object Re-Identification (ReID) aims to retrieve specific objects by integrating complementary information from multiple modalities. However, existing multi-modal ReID methods do not effectively address background interference suppression or achieve tri-modal alignment, instead focusing on pairwise feature fusion. Moreover, many current aggregation approaches suffer from high computational complexity. To address these limitations, we propose PRISM, a novel multi-modal ReID framework built upon Prompt-S6 (PS6) and semantic-aware knowledge guidance. PS6 maintains the linear complexity and strong sequence modeling capability of Mamba while enabling efficient cross-modal interaction. Leveraging these advantages, we design two key components: Semantic-Driven Token Pruning (SDTP) and Progressive Fusion Network (PFN). Parsing semantic priors from the segmentation foundation models, the SDTP then leverages these priors and applies dynamic token pruning to suppress background noise and refine feature representations. The PFN progressively aggregates multi-modal features to achieve tri-modal alignment and fully exploit modality complementarity. With the proposed modules, PRISM generates more robust multi-modal representations under complex scenarios. Extensive experiments on four multi-modal object ReID benchmarks demonstrate the effectiveness and efficiency of our approach. The source code is available at https://github.com/zw-absin/PRISM.

We introduce the Pose and Illumination agnostic Anomaly Detection (PIAD) problem, a generalization of pose-agnostic anomaly detection (PAD). Being illumination agnostic is critical, as it relaxes the assumption that training data for an object has to be acquired in the same light configuration of the query images that we want to test. Moreover, even if the object is placed within the same capture environment, being illumination agnostic implies that we can relax the assumption that the relative pose between environment light and query object has to match the one in the training data. We introduce a new dataset to study this problem, containing both synthetic and real-world examples, propose a new baseline for PIAD, and demonstrate how our baseline provides state-of-the-art results in both PAD and PIAD, not only in the new proposed dataset, but also in existing datasets that were designed for the simpler PAD problem. Project page: https://kaichen-yang.github.io/piad/.

Existing super-resolution methods exhibit limitations when applied to nighttime scenes, primarily due to their lack of adaptation to low-pair dynamic range and noise-heavy dark-light images. In response, this research introduces an innovative customized framework to simultaneously Brighten and Zoom in low-resolution images captured in low-light conditions, dubbed BrZoNet. The core method begins by feeding low-light, low-resolution images, and their corresponding ground truths into the Retinex-induced siamese decoupling network. This process yields distinct reflectance maps and illuminance maps, guided by supervision from the ground truth’s decomposition maps. Subsequently, these reflectance and illuminance maps transition into an intricate super-resolution sub-network. This sub-network employs a meticulously designed cross-layer content-aware interactor - Illumination-aware Interaction Unit(IaIU), elegantly endowed with a gating mechanism. The IaIU facilitates meaningful feature interaction between illuminance and reflectance features while effectively reducing unwanted noise. An intricate super-resolution cage is also constructed to comprehensively integrate information, ultimately resulting in the generation of high-resolution images featuring intricate details. Thorough and diverse experiments validate the superiority of the proposed BrZoNet, surpassing contemporary cutting-edge technologies by proficiently augmenting brightness and intricately recovering complex details, showcasing advancements of 7.1% in PSNR, 2.4% in SSIM, and an impressive 36.8% in LPIPS metrics.

The multi-view subspace clustering attracts much attention in recent years. Most methods follow the framework of fusing the affinity graph learned in each view. In this framework, both the fusion strategy and built graph of each view are very important. In this paper, we propose novel methods for multi-view subspace clustering to address these two aspects. On the one hand, we adopt the autoencoders with Laplacian regularization to construct the affinity graph in each view. Compared with previous work employing the autoencoders, the Laplacian term in our method can guide the learned latent representation favoring affinity extraction. Besides, we also discuss the reasons for adding Laplacian regularization. On the other hand, we propose a novel fusion strategy distinguished from the related literature. If the affinity graph of some view is not extracted well, the performance of previous fusion strategies will be seriously affected. Since our strategy can choose the best part from each affinity graph, it can overcome this limitation to some extent. Extensive experimental results on multiple benchmark data sets confirm the effectiveness of our method.

Rainy images degrade the visional performance that may bring down the accuracy of various applications. In this paper, we propose a novel densely connected network with Dense Feature Pyramid Grids Modules, called DFPGN, to solve the rain removal task. Specifically, in the proposed DFPG, there are five operations from different layers with various pathways and scales as the input of the current layer so that each layer can fuse various features from shallower and deeper ones to improve the deraining ability of the network. Extensive experiments on real and synthetic rainy images are conducted to demonstrate the proposed method achieves superior rain removal performance over state-of-the-art approaches.

We present a simple yet effective unpaired learning based image rain removal method from an unpaired set of synthetic images and real rainy images by exploring the properties of rain maps. The proposed algorithm mainly consists of a semi-supervised learning part and a knowledge distillation part. The semi-supervised part estimates the rain map and reconstructs the derained image based on the well-established layer separation principle. To facilitate rain removal, we develop a rain direction regularizer to constrain the rain estimation network in the semi-supervised learning part. With the estimated rain maps from the semi-supervised learning part, we first synthesize a new paired set by adding to rain-free images based on the superimposition model. The real rainy images and the derained results constitute another paired set. Then we develop an effective knowledge distillation method to explore such two paired sets so that the deraining model in the semi-supervised learning part is distilled. We propose two new rainy datasets, named RainDirection and Real3000, to validate the effectiveness of the proposed method. Both quantitative and qualitative experimental results demonstrate that the proposed method achieves favorable results against state-of-the-art methods in benchmark datasets and real-world images.

Deep convolutional neural networks (CNNs) have contributed to the significant progress of the single-image super-resolution (SISR) field. However, the majority of existing CNN-based models maintain high performance with massive parameters and exceedingly deeper structures. Moreover, several algorithms essentially have underused the low-level features, thus causing relatively low performance. In this article, we address these problems by exploring two strategies based on novel local wider residual blocks (LWRBs) to effectively extract the image features for SISR. We propose a cascading residual network (CRN) that contains several locally sharing groups (LSGs), in which the cascading mechanism not only promotes the propagation of features and the gradient but also eases the model training. Besides, we present another enhanced residual network (ERN) for image resolution enhancement. ERN employs a dual global pathway structure that incorporates nonlocal operations to catch long-distance spatial features from the the original low-resolution (LR) input. To obtain the feature representation of the input at different scales, we further introduce a multiscale block (MSB) to directly detect low-level features from the LR image. The experimental results on four benchmark datasets have demonstrated that our models outperform most of the advanced methods while still retaining a reasonable number of parameters.

Single image deraining is an urgent task because the degraded rainy image makes many computer vision systems fail to work, such as video surveillance and autonomous driving. So, deraining becomes important and an effective deraining algorithm is needed. In this paper, we propose a novel network based on physical model guided learning for single image deraining, which consists of three sub-networks: rain streaks network, rain-free network, and guide-learning network. The concatenation of rain streaks and rain-free image that are estimated by rain streaks network, rain-free network, respectively, is input to the guide-learning network to guide further learning and the direct sum of the two estimated images is constrained with the input rainy image based on the physical model of rainy image. Moreover, we further develop the Multi-Scale Residual Block (MSRB) to better utilize multi-scale information and it is proved to boost the deraining performance. Quantitative and qualitative experimental results demonstrate that the proposed method outperforms the state-of-the-art deraining methods. The source code will be available at https://supercong94.wixsite.com/supercong94.

In the field of multimedia, single image deraining is a basic pre-processing work, which can greatly improve the visual effect of subsequent high-level tasks in rainy conditions. In this paper, we propose an effective algorithm, called JDNet, to solve the single image deraining problem and conduct the segmentation and detection task for applications. Specifically, considering the important information on multi-scale features, we propose a Scale-Aggregation module to learn the features with different scales. Simultaneously, Self-Attention module is introduced to match or outperform their convolutional counterparts, which allows the feature aggregation to adapt to each channel. Furthermore, to improve the basic convolutional feature transformation process of Convolutional Neural Networks (CNNs), Self-Calibrated convolution is applied to build long-range spatial and inter-channel dependencies around each spatial location that explicitly expand fields-of-view of each convolutional layer through internal communications and hence enriches the output features. By designing the Scale-Aggregation and Self-Attention modules with Self-Calibrated convolution skillfully, the proposed model has better deraining results both on real-world and synthetic datasets. Extensive experiments are conducted to demonstrate the superiority of our method compared with state-of-the-art methods. The source code will be available at https://supercong94.wixsite.com/supercong94.

Rain removal is an important but challenging computer vision task as rain streaks can severely degrade the visibility of images that may make other visions or multimedia tasks fail to work. Previous works mainly focused on feature extraction and processing or neural network structure, while the current rain removal methods can already achieve remarkable results, training based on single network structure without considering the cross-scale relationship may cause information drop-out. In this paper, we explore the cross-scale manner between networks and inner-scale fusion operation to solve the image rain removal task. Specifically, to learn features with different scales, we propose a multi-sub-networks structure, where these sub-networks are fused via a cross-scale manner by Gate Recurrent Unit to inner-learn and make full use of information at different scales in these sub-networks. Further, we design an inner-scale connection block to utilize the multi-scale information and features fusion way between different scales to improve rain representation ability and we introduce the dense block with skip connection to inner-connect these blocks. Experimental results on both synthetic and real-world datasets have demonstrated the superiority of our proposed method, which outperforms over the state-of-the-art methods. The source code will be available at https://supercong94.wixsite.com/supercong94.

Image dehazing is a well-known ill-posed problem, which usually requires some image priors to make the problem well-posed. We propose an effective iteration algorithm with deep CNNs to learn haze-relevant priors for image dehazing. We formulate the image dehazing problem as the minimization of a variational model with favorable data fidelity terms and prior terms to regularize the model. We solve the variational model based on the classical gradient descent method with built-in deep CNNs so that iteration-wise image priors for the atmospheric light, transmission map and clear image can be well estimated. Our method combines the properties of both the physical formation of image dehazing as well as deep learning approaches. We show that it is able to generate clear images as well as accurate atmospheric light and transmission maps. Extensive experimental results demonstrate that the proposed algorithm performs favorably against state-of-the-art methods in both benchmark datasets and real-world images.

Recently, image dehazing has received extensive attention from researchers in vision society. Previous dehazing methods usually estimate transmissions and haze-free images in a separate way, which leads to poor image dehazing results if transmissions are incorrectly estimated. On the other hand, though some CNN-based deep networks have been developed to remove haze, their transmission estimations heavily rely on white balance. In this paper, we propose a residual type CN-N for transmission refinement rather than estimation. Benefit from its residual learning ability, we plug the network in solving an optimization problem, which is able to improve the refinement results through jointly estimating transmissions and clean images in a single framework. Experimental results of synthetic and real-world images demonstrate the superiority and efficiency of our proposed framework, compared to many state-of-the-art methods.

In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatial-temporal semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.

Deblurring images with outliers has attracted considerable attention recently. However, existing algorithms usually involve complex operations which increase the difficulty of blur kernel estimation. In this paper, we propose a simple yet effective blind image deblurring algorithm to handle blurred images with outliers. The proposed method is motivated by the observation that outliers in the blurred images significantly affect the goodness-of-fit in function approximation. Therefore, we propose an algorithm to model the data fidelity term so that the outliers have little effect on kernel estimation. The proposed algorithm does not require any heuristic outlier detection step, which is critical to the state-of-the-art blind deblurring methods for images with outliers. We analyze the relationship between the proposed algorithm and other blind deblurring methods with outlier handling and show how to estimate intermediate latent images for blur kernel estimation principally. We show that the proposed method can be applied to generic image deblurring as well as non-uniform deblurring. Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art blind image deblurring methods on both synthetic and real-world images.

Solving blind image deblurring usually requires defining a data fitting function and image priors. While existing algorithms mainly focus on developing image priors for blur kernel estimation and non-blind deconvolution, only a few methods consider the effect of data fitting functions. In contrast to the state-of-the-art methods that use a single or a fixed data fitting term, we propose a data-driven approach to learn effective data fitting functions from a large set of motion blurred images with the associated ground truth blur kernels. The learned data fitting function facilitates estimating accurate blur kernels for generic scenes and domain-specific problems with corresponding image priors. In addition, we extend the learning approach for data fitting function to latent image restoration and nonuniform deblurring. Extensive experiments on challenging motion blurred images demonstrate the proposed algorithm performs favorably against the state-of-the-art methods.

Image matching is a challenging problem as different views often undergo significant appearance changes caused by deformation, abrupt motion, and occlusion. In this paper, we explore features extracted from convolutional neural networks to help the estimation of image matching so that dense pixel correspondence can be built. As the deep features are able to describe the image structures, the matching method based on these features is able to match across different scenes and/or object appearances. We analyze the deep features and compare them with other robust features, e.g., SIFT. Extensive experiments on 5 datasets demonstrate the proposed algorithm performs favorably against the state-of-the-art methods in terms of visually matching quality and accuracy.

-regularized prior based on intensity and gradient for text image deblurring. The proposed image prior is based on distinctive properties of text images, with which we develop an efficient optimization algorithm to generate reliable intermediate results for kernel estimation. The proposed algorithm does not require any heuristic edge selection methods, which are critical to the state-of-the-art edge-based deblurring methods. We discuss the relationship with other edge-based deblurring methods and present how to select salient edges more principally. For the final latent image restoration step, we present an effective method to remove artifacts for better deblurred results. We show the proposed algorithm can be extended to deblur natural images with complex scenes and low illumination, as well as non-uniform deblurring. Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art image deblurring methods.

Being one of the most effective methods, Alternating Direction Method (ADM) has been extensively studied in numerical analysis for solving linearly constrained convex program. However, there are few studies focusing on the convergence property of ADM under nonconvex framework though it has already achieved well-performance on applying to various nonconvex tasks. In this paper, a linearized algorithm with penalization is proposed on the basis of ADM for solving nonconvex and nonsmooth optimization. We start from analyzing the convergence property for the classical constrained problem with two variables and then establish a similar result for multi-block case. To demonstrate the effectiveness of our proposed algorithm, experiments with synthetic and real-world data have been conducted on specific applications in signal and image processing.

Object motion blur is a challenging problem as the foreground and the background in the scenes undergo different types of image degradation due to movements in various directions and speed. Most object motion deblurring methods address this problem by segmenting blurred images into regions where different kernels are estimated and applied for restoration. Segmentation on blurred images is difficult due to ambiguous pixels between regions, but it plays an important role for object motion deblurring. To address these problems, we propose a novel model for object motion deblurring. The proposed model is developed based on a maximum a posterior formulation in which soft-segmentation is incorporated for object layer estimation. We propose an efficient algorithm to jointly estimate object segmentation and camera motion where each layer can be deblurred well under the guidance of the soft-segmentation. Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art object motion deblurring methods on challenging scenarios.

Estimating blur kernels from real world images is a challenging problem as the linear image formation assumption does not hold when significant outliers, such as saturated pixels and non-Gaussian noise, are present. While some existing non-blind deblurring algorithms can deal with outliers to a certain extent, few blind deblurring methods are developed to well estimate the blur kernels from the blurred images with outliers. In this paper, we present an algorithm to address this problem by exploiting reliable edges and removing outliers in the intermediate latent images, thereby estimating blur kernels robustly. We analyze the effects of outliers on kernel estimation and show that most state-of-the-art blind deblurring methods may recover delta kernels when blurred images contain significant outliers. We propose a robust energy function which describes the properties of outliers for the final latent image restoration. Furthermore, we show that the proposed algorithm can be applied to improve existing methods to deblur images with outliers. Extensive experiments on different kinds of challenging blurry images with significant amount of outliers demonstrate the proposed algorithm performs favorably against the state-of-the-art methods.

Informative and discriminative feature descriptors are vital in qualitative and quantitative shape analysis for a large variety of graphics applications. Conventional feature descriptors primarily concentrate on discontinuity of certain differential attributes at different orders that naturally give rise to their discriminative power in depicting point, line, small patch features, etc. This paper seeks novel strategies to define generalized, user-specified features anywhere on shapes. Our new region-based feature descriptors are constructed primarily with the powerful spectral graph wavelets (SGWs) that are both multi-scale and multi-level in nature, incorporating both local (differential) and global (integral) information. To our best knowledge, this is the first attempt to organize SGWs in a hierarchical way and unite them with the bi-harmonic diffusion field towards quantitative region-based shape analysis. Furthermore, we develop a local-to-global shape feature detection framework to facilitate a host of graphics applications, including partial matching without point-wise correspondence, coarse-to-fine recognition, model recognition, etc. Through the extensive experiments and comprehensive comparisons with the state-of-the-art, our framework has exhibited many attractive advantages such as being geometry-aware, robust, discriminative, isometry-invariant, etc.

In this paper, we incorporate sparse coding and orthogonal dictionary learning into a unified framework, named orthogonal sparse coding (OSC), for robust visual tracking. Different from previous tracking methods, which often use redundant dictionaries, OSC enforces an orthogonality constraint in the dictionary learning step to adaptively capture the structures of the video sequences. Moreover, a ℓ0norm regularizer is introduced in OSC formulation to address the severe noise problems, illumination changes, and occlusions in real world videos. As a nontrivial byproduct, we develop an efficient numerical solver to address the optimization issues of our OSC model. Experimental results on various challenging video sequences show that the proposed method achieves better performance both on accuracy and speed compared to proposed state-of-the-art methods.

Visual tracking is a fundamental task in computer vision. Although many efforts have been made in the past decades, it is still challenging to handle the complex factors in real world tracking scenarios. Ranking methods have shown their power on different data analysis tasks. However, we can not directly utilize this technique on sequential data for tracking. This is because a single ranking model cannot simultaneously reveal both the spatial and the temporal information. In this paper, we propose a novel discriminative sequential ranking (DSR) method to build appearance model for robust visual tracking. Our method can successfully handle both spatial and temporal variations by the coupled ranking processes. Specifically, the spatial process provides a target probability to reflects the intrinsic structure of the object at current frame. Meanwhile, the temporal process provides a background probability (guided by the sequential information) to stably describe the background appearance, which makes our tracker robust for background clutter. Experimental evaluations on the benchmark database with 50 challenging videos confirm that our method outperforms many other state-of-the-art tracking algorithms.

Visual tracking is a fundamental task in computer vision. Although many efforts have been made in the past decades, it is still challenging to handle the complex factors in real world tracking scenarios. Ranking methods have shown their power on different data analysis tasks. However, we can not directly utilize this technique on sequential data for tracking. This is because a single ranking model cannot simultaneously reveal both the spatial and the temporal information. In this paper, we propose a novel discriminative sequential ranking (DSR) method to build appearance model for robust visual tracking. Our method can successfully handle both spatial and temporal variations by the coupled ranking processes. Specifically, the spatial process provides a target probability to reflects the intrinsic structure of the object at current frame. Meanwhile, the temporal process provides a background probability (guided by the sequential information) to stably describe the background appearance, which makes our tracker robust for background clutter. Experimental evaluations on the benchmark database with 50 challenging videos confirm that our method outperforms many other state-of-the-art tracking algorithms.

We propose a simple yet effective L0-regularized prior based on intensity and gradient for text image deblurring. The proposed image prior is motivated by observing distinct properties of text images. Based on this prior, we develop an efficient optimization method to generate reliable intermediate results for kernel estimation. The proposed method does not require any complex filtering strategies to select salient edges which are critical to the state-of-the-art deblurring algorithms. We discuss the relationship with other deblurring algorithms based on edge selection and provide insight on how to select salient edges in a more principled way. In the final latent image restoration step, we develop a simple method to remove artifacts and render better deblurred images. Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art text image deblurring methods. In addition, we show that the proposed method can be effectively applied to deblur low-illumination images.

Visual tracking is a fundamental task in computer vision. In this paper, we propose an incremental robust local dictionary learning framework to address this problem. We first initialize a dictionary using local low-rank features to represent the appearance subspace for the object. In this way, each candidate can be modeled by the sparse linear representation of the learnt dictionary. Then by incrementally updating the local dictionary and learning sparse representation for the candidate, we build a robust online object tracking system. Compared with conventional methods, which directly use corrupted observations to form the dictionary, our local low-rank features based dictionary successfully remove occlusions and exactly represent the intrinsic structure of the object. Furthermore, in contrast to the traditional holistic dictionary, the local low-rank features based dictionary contain abundant partial information and spatial information. Experimental results on challenging image sequences show that our method consistently outperforms several state-of-the-art methods.

In this paper, a novel subspace learning algorithm is proposed for robust visual tracking. Different from conventional sub-space based trackers, which first estimate the dimension of the subspace and then pursuit its basis to construct the subspace projection in appearance model, our method directly learns a low-rank projection with known ranks as subspace dimension to model the subspace structure for visual tracking. Under particle filter tracking framework, an online scheme is developed to incrementally pursue the optimum projection and the candidate with the minimal reconstruction error is selected to deliver the tracking information to the next frame and pursue the projection. The columns of the projection defined in the latent feature space are a set of redundant basis, treating an observation as its coefficient. As a result, the low-rank property of the pursued optimum projection can exactly reveal the intrinsic low-dimensional structure of the global feature space, contributing to the high precision of capturing appearance changes. Experiments on several challenging image sequences demonstrate that our tracker performs excellently against several state-of-the-art trackers.

Blind image deblurring, i.e., estimating a blur kernel from a single input blurred image is a severely ill-posed problem. In this paper, we show how to effectively apply low rank prior to blind image deblurring and then propose a new algorithm which combines salient edges and low rank prior. Salient edges provide reliable edge information for kernel estimation, while low rank prior provides data-authentic priors for the latent image. When estimating the kernel, the salient edges are extracted from an intermediate latent image solved by combining the predicted edges and low rank prior, which help preserve more useful edges than previous deconvolution methods do. By solving the blind image deblurring problem in this fashion, high-quality blur kernels can be obtained. Extensive experiments testify to the superiority of the proposed method over state-of-the-art algorithms, both qualitatively and quantitatively.

Benefiting from its effectiveness in subspace segmentation, low-rank representation (LRR) and its variations have many applications in computer vision and pattern recognition, such as motion segmentation, image segmentation, saliency detection, and semisupervised learning. It is known that the standard LRR can only work well under the assumption that all the subspaces are independent. However, this assumption cannot be guaranteed in real-world problems. This paper addresses this problem and provides an extension of LRR, named structure-constrained LRR (SC-LRR), to analyze the structure of multiple disjoint subspaces, which is more general for real vision data. We prove that the relationship of multiple linear disjoint subspaces can be exactly revealed by SC-LRR, with a predefined weight matrix. As a nontrivial byproduct, we also illustrate that SC-LRR can be applied for semisupervised learning. The experimental results on different types of vision problems demonstrate the effectiveness of our proposed method.

Many problems in statistics and machine learning (e.g., probabilistic graphical model, feature extraction, clustering and classification, etc) can be (re)formulated as linearly constrained separable convex programs. The traditional alternating direction method (ADM) or its linearized version (LADM) is for the two-variable case and \emphcannot be naively generalized to solve the multi-variable case. In this paper, we propose LADM with parallel splitting and adaptive penalty (LADMPSAP) to solve multi-variable separable convex programs efficiently. When all the component objective functions have bounded subgradients, we obtain convergence results that are stronger than those of ADM and LADM, e.g., allowing the penalty parameter to be unbounded and proving the \emphsufficient and necessary conditions for global convergence. We further propose a simple optimality measure and reveal the convergence \emphrate of LADMPSAP in an ergodic sense. For programs with extra convex set constraints, we devise a practical version of LADMPSAP for faster convergence. LADMPSAP is particularly suitable for sparse representation and low-rank recovery problems because its subproblems have closed form solutions and the sparsity and low-rankness of the iterates can be preserved during the iteration. It is also \emphhighly parallelizable and hence fits for parallel or distributed computing. Numerical experiments testify to the speed and accuracy advantages of LADMPSAP.

Geometry images are important representations of 3D geometry models. Smooth geometry images contributes to low approximation error and high image compressibility. We present an adapted parameterization method to generate a smooth geometry image. It is quite challenging to directly modify the parameter domain to smooth geometry images. Our novel idea is that we use an indirect way to construct a resulting parameter domain according to the desired geometry image. We first move image pixels according to the current parameter domain to decrease the local linear error. Then we formulate a relationship between the moved image pixels and the current parameter domain. Finally, we use the relationship to update the parameter domain by restituting the image pixels to their original positions. The process will continue until the local linear error is less than a given threshold or the number of iterations is larger than a given threshold. Experimental results illustrate that geometry images generated by our method have low linear errors and low approximation errors under different sampling resolutions.

How to detect visual salient regions is a challenging problem in computer vision. Recently, saliency detection methods that use boundaries or convex hulls under Bayesian framework have attracted lots of attention. Although these methods achieve state-of-the-art results, there still exist some limitations, e.g., the background will get highlighted when the initial convex hulls are not good enough. This paper presents a new algorithm that retains the advantages of such saliency maps while overcoming their shortcomings. First, the initial convex hull is improved by the image matting model which can be efficiently solved by an edge-preserving filter. Second, a more accurate prior map can be obtained by the improved convex hull. Third, the final convex hull is further refined by an edge-preserving filter to compute the observation likelihood. Finally, the Bayesian framework is employed to compute the saliency map. Extensive experiments compared with state-of-the-art saliency detection algorithms demonstrate the effectiveness of our method.

Conventional saliency detection approaches are human fixation detection and single dominant region detection. However, real-world photographs usually consist of multiple dominant regions. We propose a saliency detection method with the aim to highlight objects as a whole and distinguish objects with different saliency levels. It combines the bottom-up approach and top-down approach via two nested levels of hierarchical segmentations - the coarse level objects and fine level details. We first calculate a preliminary saliency on the fine patches with a random walk model. Then a location cue and an object-level cue are fused to refine the preliminary saliency to emphasize the objects against the background. At last, the object-level saliency map is synthesized via a heat diffusion process restricted by the coarse level patches to enhance object saliency and distinguish saliency between different objects. Extensive evaluation on a publicly available database verifies that our method outperforms the state-of-the-art algorithms.

Blind image deblurring is a challenging problem in computer vision and image processing. In this paper, we propose a newl0-regularized approach to estimate a blur kernel from a single blurred image by regularizing the sparsity property of natural images. Furthermore, by introducing an adaptive structure map in the deblurring process, our method is able to restore useful salient edges for kernel estimation. Finally, we propose an efficient algorithm which can solve the proposed model efficiently. Extensive experiments compared with state-of-the-art blind deblurring methods demonstrate the effectiveness of the proposed method.

The extraction and classification of multitype (point, curve, patch) features on manifolds are extremely challenging, due to the lack of rigorous definition for diverse feature forms. This paper seeks a novel solution of multitype features in a mathematically rigorous way and proposes an efficient method for feature classification on manifolds. We tackle this challenge by exploring a quasi-harmonic field (QHF) generated by elliptic PDEs, which is the stable state of heat diffusion governed by anisotropic diffusion tensor. Diffusion tensor locally encodes shape geometry and controls velocity and direction of the diffusion process. The global QHF weaves points into smooth regions separated by ridges and has superior performance in combating noise/holes. Our method's originality is highlighted by the integration of locally defined diffusion tensor and globally defined elliptic PDEs in an anisotropic manner. At the computational front, the heat diffusion PDE becomes a linear system with Dirichlet condition at heat sources (called seeds). Our new algorithms afford automatic seed selection, enhanced by a fast update procedure in a high-dimensional space. By employing diffusion probability, our method can handle both manufactured parts and organic objects. Various experiments demonstrate the flexibility and high performance of our method.

Subspace clustering and feature extraction are two of the most commonly used unsupervised learning techniques in computer vision and pattern recognition. State-of-the-art techniques for subspace clustering make use of recent advances in sparsity and rank minimization. However, existing techniques are computationally expensive and may result in degenerate solutions that degrade clustering performance in the case of insufficient data sampling. To partially solve these problems, and inspired by existing work on matrix factorization, this paper proposes fixed-rank representation (FRR) as a unified framework for unsupervised visual learning. FRR is able to reveal the structure of multiple subspaces in closed-form when the data is noiseless. Furthermore, we prove that under some suitable conditions, even with insufficient observations, FRR can still reveal the true subspace memberships. To achieve robustness to outliers and noise, a sparse regularizer is introduced into the FRR framework. Beyond subspace clustering, FRR can be used for unsupervised feature extraction. As a non-trivial byproduct, a fast numerical solver is developed for FRR. Experimental results on both synthetic data and real applications validate our theoretical analysis and demonstrate the benefits of FRR for unsupervised visual learning.

This paper studies the problem of learning robust regression for real world head pose estimation. The performance and applicability of traditional regression methods in real world head pose estimation are limited by a lack of robustness to outlying or corrupted observations. By introducing low- rank and sparse regularizations, we propose a novel regression method, named Convex Regularized Sparse Regression (CRSR), for simultaneously removing the noise and outliers from the training data and learning the regression between image features and pose angles. We verify the efficiency of the proposed robust regression method with extensive experiments on real data, demonstrating lower error rates and efficiency than existing methods.

Many machine learning and signal processing problems can be formulated as linearly constrained convex programs, which could be efficiently solved by the alternating direction method (ADM). However, usually the subproblems in ADM are easily solvable only when the linear mappings in the constraints are identities. To address this issue, we propose a linearized ADM (LADM) method by linearizing the quadratic penalty term and adding a proximal term when solving the subproblems. For fast convergence, we also allow the penalty to change adaptively according a novel update rule. We prove the global convergence of LADM with adaptive penalty (LADMAP). As an example, we apply LADMAP to solve low-rank representation (LRR), which is an important subspace clustering technique yet suffers from high computation cost. By combining LADMAP with a skinny SVD representation technique, we are able to reduce the complexity $O(n^3)$ of the original ADM based method to $O(rn^2)$, where $r$ and $n$ are the rank and size of the representation matrix, respectively, hence making LRR possible for large scale applications. Numerical experiments verify that for LRR our LADMAP based methods are much faster than state-of-the-art algorithms.

We propose a novel curvature-aware simplification technique for point-sampled geometry based on the locally optimal projection (LOP) operator. Our algorithm includes two new developments. First, a weight term related to surface variation at each point is introduced to the classic LOP operator. It produces output points with a spatially adaptive distribution. Second, for speeding up the convergence of our method, an initialization process is proposed based on geometry-aware stochastic sampling. Owing to the initialization, the relaxation process achieves a faster convergence rate than those initialized by uniform sampling. Our simplification method possesses a number of distinguishing features. In particular, it provides resilience to noise and outliers, and an intuitively controllable distribution of simplification. Finally, we show the results of our approach with publicly available point cloud data, and compare the results with those obtained using previous methods. Our method outperforms these methods on raw scanned data.

Optical flow estimation is still an important task in computer vision with many interesting applications. However, the results obtained by most of the optical flow techniques are affected by motion discontinuities or illumination changes. In this paper, we introduce a brightness correction field combined with a gradient constancy constraint to reduce the sensibility to brightness changes between images to be estimated. The advantage of this brightness correction field is its simplicity in terms of computational complexity and implementation. By analyzing the deficiencies of the traditional total variation regularization term in weakly textured areas, we also adopt a structure-adaptive regularization based on the robust Huber norm to preserve motion discontinuities. Finally, the proposed energy functional is minimized by solving its corresponding Euler-Lagrange equation in a more effective multi-resolution scheme, which integrates the twice downsampling strategy with a support-weight median filter. Numerous experiments show that our method is more effective and produces more accurate results for optical flow estimation.

We present a novel algorithm to reconstruct curves with self-intersections and multiple parts from unorganized strip-shaped points, which may have different local shape scales and sampling densities. We first extract an initial curve, a graph composed of polylines, to model the different structures of the points. Then a least-squares optimization is used to improve the geometric approximation. The initial curve is extracted in three steps: anisotropic farthest point sampling with an adaptable sphere, graph construction followed by non-linear region identification, and edge refinement. Our algorithm produces faithful results for points sampled from non-simple curves without pre-segmenting them. Experiments on many simulated and real data demonstrate the efficiency of our method, and more faithful curves are reconstructed compared to other existing methods.

We present an algorithm for curve skeleton extraction via Laplacian-based contraction. Our algorithm can be applied to surfaces with boundaries, polygon soups, and point clouds. We develop a contraction operation that is designed to work on generalized discrete geometry data, particularly point clouds, via local Delaunay triangulation and topological thinning. Our approach is robust to noise and can handle moderate amounts of missing data, allowing skeleton-based manipulation of point clouds without explicit surface reconstruction. By avoiding explicit reconstruction, we are able to perform skeleton-driven topology repair of acquired point clouds in the presence of large amounts of missing data. In such cases, automatic surface reconstruction schemes tend to produce incorrect surface topology. We show that the curve skeletons we extract provide an intuitive and easy-to-manipulate structure for effective topology modification, leading to more faithful surface reconstruction.

One major goal of mesh parameterization is to minimize the conformal distortion. Measured boundary parameterizations focus on lowering the distortion by setting the boundary free with the help of distance from a center vertex to all the boundary vertices. Hence these parameterizations strongly depend on the determination of the center vertex. In this paper, we introduce two methods to determine the center vertex automatically. Both of them can be used as necessary supplements to the existing measured boundary methods to minimize the common artifacts as a result of the obscure choice of the center vertex. In addition, we propose a simple and fast measured boundary parameterization method based on the Poisson’s equation. Our new approach generates less conformal distortion than the fixed boundary methods. It also generates more regular domain boundaries than other measured boundary methods. Moreover, it offers a good tradeoff between computation costs and conformal distortion compared with the fast and robust angle based flattening (ABF++).

In this paper, we propose a novel triangle mesh denoising method based on the differential coordinates. The proposed approach consists of the application of the mean filter to differential coordinates of the mesh and the reconstruction of mesh vertices' Cartesian coordinates to make them fit to the modified differential coordinates. The presented method is simple, stable and able to effectively remove large noise. Experimental results demonstrate that the proposed Mesh Mean Filter does not cause surface shrinkage and shape distortion during the denoising process, and preserves geometric detail features to a certain extent.

Two algorithms for evaluation of regular quad-mesh interpolatory subdivision surfaces are proposed. Algorithms are designed based on the parametric m-ary decomposition and construction of matrix sequence. The weights of the control points on the initial mesh can be obtained, through direct computation of the basic function values by multiplying the finite matrix sequence corresponding to the decomposition number sequence. Algorithm-I is based on 2D subdivision masks while the other is based on tensor-product. Numerical experiments show that algorithms are efficient with low storage cost.

In this paper, we present a method to simultaneously blend the corner of three coordinate planes with G2 continuity by cubic algebraic spline surfaces. This method is based on space partition and algebraic splines.