Deriving from the gradient vector of a generative model of local features, Fisher vector coding (FVC) has been identified as an effective coding method for image classification. Most, if not all, FVC implementations employ the Gaussian mixture model (GMM) as the generative model for local features. However, the representative power of a GMM can be limited because it essentially assumes that local features can be characterized by a fixed number of feature prototypes, and the number of prototypes is usually small in FVC. To alleviate this limitation, in this work, we break the convention which assumes that a local feature is drawn from one of a few Gaussian distributions. Instead, we adopt a compositional mechanism which assumes that a local feature is drawn from a Gaussian distribution whose mean vector is composed as a linear combination of multiple key components, and the combination weight is a latent random variable. In doing so we greatly enhance the representative power of the generative model underlying FVC. To implement our idea, we design two particular generative models following this compositional approach. In our first model, the mean vector is sampled from the subspace spanned by a set of bases and the combination weight is drawn from a Laplace distribution. In our second model, we further assume that a local feature is composed of a discriminative part and a residual part. As a result, a local feature is generated by the linear combination of discriminative part bases and residual part bases. The decomposition of the discriminative and residual parts is achieved via the guidance of a pre-trained supervised coding method. By calculating the gradient vector of the proposed models, we derive two new Fisher vector coding strategies. The first is termed Sparse Coding-based Fisher Vector Coding (SCFVC) and can be used as the substitute of traditional GMM based FVC. The second is termed Hybrid Sparse Coding-based Fisher vector coding (HSCFVC) since it combines the merits of both pre-trained supervised coding methods and FVC. Using pre-trained Convolutional Neural Network (CNN) activations as local features, we experimentally demonstrate that the proposed methods are superior to traditional GMM based FVC and achieve state-of-the-art performance in various image classification tasks.

Graph matching has been widely used in both image processing and computer vision domain due to its powerful performance for structural pattern representation. However, it poses three challenges to image sparse feature matching: 1) the combinatorial nature limits the size of the possible matches; 2) it is sensitive to outliers because its objective function prefers more matches; and 3) it works poorly when handling many-to-many object correspondences, due to its assumption of one single cluster of true matches. In this paper, we address these challenges with a unified framework called density maximization (DM), which maximizes the values of a proposed graph density estimator both locally and globally. DM leads to the integration of feature matching, outlier elimination, and cluster detection. Experimental evaluation demonstrates that it significantly boosts the true matches and enables graph matching to handle both outliers and many-to-many object correspondences. We also extend it to dense correspondence estimation and obtain large improvement over the state-of-the-art methods. We further demonstrate the usefulness of our methods using three applications: 1) instance-level image retrieval; 2) mask transfer; and 3) image enhancement.

Volume up scaling enlarges the size of a volume to make feature analysis more accurate and efficient. Linear interpolation, widely used in volume up scaling, result in jagged artifacts around features and losses of high-frequency components. Based on the example-based up scaling framework, this paper presents a new high-quality volume up scaling technique, predicting the high-frequency components by searching for the best matched patch in the input volume. As each slice can be taken as an image, the existing image up scaling technique based on the local self-similarity assumption can be directly applied to achieve slice up scaling. We further validate that the local self-similarity assumption is still valid for 3D volumes, and we extend this technique to 3D volume up scaling, i.e., isotropic volume up scaling. We compare our volume up scaling technique with traditional linear and cubic spline interpolations, and demonstrate that our method can generate a higher quality volume with better shape and details preserved. The proposed volume up scaling technique is well suitable for legacy low-resolution volumes to improve their visual qualities in visualization and analysis.

Graph matching has been widely used in various applications in computer vision due to its powerful performance. However, it poses three challenges to image sparse feature matching: (1) The combinatorial nature limits the size of the possible matches, (2) It is sensitive to outliers because the objective function prefers more matches, (3) It works poorly when handling many-to-many object correspondences, due to its assumption of one single cluster for each graph. In this paper, we address these problems with a unified framework-Density Maximization. We propose a graph density local estimator (DLE) to measure the quality of matches. Density Maximization aims to maximize the DLE values both locally and globally. The local maximization of DLE finds the clusters of nodes as well as eliminates the outliers. The global maximization of DLE efficiently refines the matches by exploring a much larger matching space. Our Density Maximization is orthogonal to specific graph matching algorithms. Experimental evaluation demonstrates that it significantly boosts the true matches and enables graph matching to handle both outliers and many-to-many object correspondences.

Abstract The aim of removing camera shake is to estimate a sharp version x from a shaken image y when the blur kernel k is unknown. Recent research on this topic evolved through two paradigms called and . only solves for k by marginalizing the image prior, while recovers both x and k by selecting the mode of the posterior distribution. This paper first systematically analyses the latent limitations of these two estimators through Bayesian analysis. We explain the reason why it is so difficult for image statistics to solve the previously reported failure. Then we show that the leading methods, which depend on efficient prediction of large step edges, are not robust to natural images due to the diversity of edges. , although much more robust to diverse edges, is constrained by two factors: the prior variation over different images, and the ratio between image size and kernel size. To overcome these limitations, we introduce an inter‐scale prior prediction scheme and a principled mechanism for integrating the sharpening filter into . Both qualitative results and extensive quantitative comparisons demonstrate that our algorithm outperforms state‐of‐the‐art methods.

Blind motion deblurring estimates a sharp image from a motion blurred image without the knowledge of the blur kernel. Although significant progress has been made on tackling this problem, existing methods, when applied to highly diverse natural images, are still far from stable. This paper focuses on the robustness of blind motion deblurring methods toward image diversity-a critical problem that has been previously neglected for years. We classify the existing methods into two schemes and analyze their robustness using an image set consisting of 1.2 million natural images. The first scheme is edge-specific, as it relies on the detection and prediction of large-scale step edges. This scheme is sensitive to the diversity of the image edges in natural images. The second scheme is nonedge-specific and explores various image statistics, such as the prior distributions. This scheme is sensitive to statistical variation over different images. Based on the analysis, we address the robustness by proposing a novel nonedge-specific adaptive scheme (NEAS), which features a new prior that is adaptive to the variety of textures in natural images. By comparing the performance of NEAS against the existing methods on a very large image set, we demonstrate its advance beyond the state-of-the-art.

To recover a sharp version from a blurred image is a long-standing inverse problem. In this paper, we analyze the research on this topic both theoretically and experimentally through three paradigms: 1) the deterministic filter; 2) Bayesian estimation; and 3) the conjunctive deblurring algorithm (CODA), which performs the deterministic filter and Bayesian estimation in a conjunctive manner. We point out the weaknesses of the deterministic filter and unify the limitation latent in two kinds of Bayesian estimators. We further explain why the CODA is able to handle quite large blurs beyond Bayesian estimation. Finally, we propose a novel method to overcome several unreported limitations of the CODA. Although extensive experiments demonstrate that our method outperforms state-of-the-art methods with a large margin, some common problems of image deblurring still remain unsolved and should attract further research efforts.

Lighting design is a complex, but fundamental, problem in many fields. In volume visualization, direct volume rendering generates an informative image without external lighting, as each voxel itself emits radiance. However, external lighting further improves the shape and detail perception of features, and it also determines the effectiveness of the communication of feature information. The human visual system is highly effective in extracting structural information from images, and to assist it further, this paper presents an approach to structure-aware automatic lighting design by measuring the structural changes between the images with and without external lighting. Given a transfer function and a viewpoint, the optimal lighting parameters are those that provide the greatest enhancement to structural information - the shape and detail information of features are conveyed most clearly by the optimal lighting parameters. Besides lighting goodness, the proposed metric can also be used to evaluate lighting similarity and stability between two sets of lighting parameters. Lighting similarity can be used to optimize the selection of multiple light sources so that different light sources can reveal distinct structural information. Our experiments with several volume data sets demonstrate the effectiveness of the structure-aware lighting design approach. It is well suited to use by novices as it requires little technical understanding of the rendering parameters associated with direct volume rendering.

Macroblock level Horizontal Spatial Prediction(HSP) based intra frame coding scheme for High Dimension(HD) video sequences was proposed in this paper. According to the correlation experiment on HD sequences, most HD pictures have the stronger horizontal spatial correlation than the vertical spatial correlation, about 2dB stronger. This phenomena drop a valuable hint to us that the horizontal spatial prediction can be used in HD video intra coding without considering the vertical spatial prediction. An adaptive divide and predict intra frame coding scheme has been proposed by Piao which has the similar idea. But this method divided the whole picture into several parts which is not conform to the conventional macroblock based video coding framework and it has the high computation complexity in motion estimation procedure.

In this paper, a fast RDO (rate-distortion optimization) quantization algorithm for H.264/AVC is proposed. In this algorithm, the searching space of level adjustments is reduced by filtering the input quantized coefficients in a hierarchical way. The well quantized coefficients is first filtered out, and then the RD tradeoff of each level adjustment to each of the rest coefficients is examined to select some good candidates with their associated level adjustments. Finally these good candidates are combined to find the best combination of level adjustments which gives the minimal rate-distortion cost. Furthermore, a fast rate estimation technique is adopted to save the rate-distortion estimation time. Experimental results show that about 44% quantization time on average can be saved at the cost of negligible PSNR loss compared with RDO quantization algorithm implemented in JM.

Content-based video retrieval systems are desired to fast and accurately find the nearest-neighbors of user input examples from very large video datasets. This poses a great challenge since exhaustive and redundant computation of similarities is required. Cluster based index approaches can be used to address this problem, but the similarity computation and clustering methods for videos are very time-consuming, thus preventing it from indexing very large video datasets. In this paper, we propose the Hierarchical Frame Affinity Group (HFAG), which is a hierarchy of frame clusters built using affinity propagation (AP) method, to represent video clusters. Our proposed video similarity metric and AP method guarantee the high performance of forming HFAG. We then build the cluster-based index structure to support retrieval of the nearest-neighbors of video sequences. The experiments on real large video datasets prove the effectiveness and efficiency of our approach.

Kernel estimate errors and image noise are major causes of visual artifacts in image motion deblurring. We propose an inter-scale non-blind image motion deblurring approach that significantly reduces those artifacts. We use Gaussian Scale Mixture Field of Experts (GSM FOE) model as image prior. The inter-scale smoothness constraint is adopted to suppress the ringing artifacts. In each scale, image details are recovered by the residual deconvolution and the cross bilateral filter (CBF). We further propose a std-controlled CBF to denoise the result. The experimental results are much better than those of previous methods.

Traditional non-blind motion deblurring methods are sensitive to kernel estimate errors and image noise, thus suffering from either ringing artifacts, enlarged image noise, or over-smoothed image details. We introduce a robust non-blind deblurring algorithm that produces high quality results even from many challenging images with noisy kernels. We adopt the Gaussian Scale Mixture Fields of Experts (GSM FOE) model and the smoothness constraint as image prior, and use the iterative re-weight least-square (IRLS) algorithm to produce the temporal result. The residual deconvolution suite is used to restore the lost image details. We denoise the result using our std-controlled cross bilateral filter. The experimental results are much better than those of previous approaches.

In this paper, we study the properties of the bilateral pyramid, based on which a novel image representation-the bilateral wavelet pyramid (BWP) is proposed. The BWP provides a multiresolution technique to manipulate image variances simultaneously in radiometric, spatial and frequency domains. Application to texture retrieval is discussed to demonstrate the effectiveness of the BWP.

Recently, there has been a great deal of interest in statistics of natural images in linear domain. However, a detail study of image statistics in non-linear domain on a very large dataset is still missing. In this paper, we analyze natural images' statistics in ldquobilateral domainrdquo on different sub bands which are generated by the classical bilateral filter. Compared with those in linear domain, the statistics in bilateral domain are significantly superior in generalization, de-correlation, and local feature extraction. We also observe some entirely new but interesting features.

Motion-detection only games have inherent limitations on game experience in that the systems cannot identify the player's existence and identity. A way of improvement is by introducing information such as a player's face or head into the system. We designed and implemented two game prototypes that apply real-time face position information as intrinsic elements of gameplay to enhance game experience. The first prototype augmented a typical motion-detection-based game. Face information was designed to enhance the sense of presence and role-playing. In the second prototype, face tracking is applied as a new axis of control in a First Person Shooter (FPS) game.Although Face detection and tracking technology has started utilizing in game scenarios, there was little systematic research on how user experience is leveraged by applying face information to video games. The results of our user tests on comparing camera-based video games with and without face tracking demonstrated that using face position information can effectively enhance presence and role-playing. In addition, an intuitive control that augmented by face-tracking in the FPS game also got positive feedbacks from the test.

We present a visual simulation technique called appearance manifolds for modeling the time-variant surface appearance of a material from data captured at a single instant in time. In modeling time-variant appearance, our method takes advantage of the key observation that concurrent variations in appearance over a surface represent different degrees of weathering. By reorganizing these various appearances in a manner that reveals their relative order with respect to weathering degree, our method infers spatial and temporal appearance properties of the material's weathering process that can be used to convincingly generate its weathered appearance at different points in time. Results with natural non-linear reflectance variations are demonstrated in applications such as visual simulation of weathering on 3D models, increasing and decreasing the weathering of real objects, and material transfer with weathering effects.