This paper addresses the problem of recognizing and removing shadows from monochromatic natural images from a learning-based perspective. Without chromatic information, shadow recognition and removal are extremely challenging in this paper, mainly due to the missing of invariant color cues. Natural scenes make this problem even harder due to the complex illumination condition and ambiguity from many near-black objects. In this paper, a learning-based shadow recognition and removal scheme is proposed to tackle the challenges above-mentioned. First, we propose to use both shadow-variant and invariant cues from illumination, texture, and odd order derivative characteristics to recognize shadows. Such features are used to train a classifier via boosting a decision tree and integrated into a conditional random field, which can enforce local consistency over pixel labels. Second, a Gaussian model is introduced to remove the recognized shadows from monochromatic natural scenes. The proposed scheme is evaluated using both qualitative and quantitative results based on a novel database of hand-labeled shadows, with comparisons to the existing state-of-the-art schemes. We show that the shadowed areas of a monochromatic image can be accurately identified using the proposed scheme, and high-quality shadow-free images can be precisely recovered after shadow removal.

Deep neural networks have achieved remarkable performance in both image classification and object detection problems, at the cost of a large number of parameters and computational complexity. In this work, we show how to reduce the redundancy in these parameters using a sparse decomposition. Maximum sparsity is obtained by exploiting both inter-channel and intra-channel redundancy, with a fine-tuning step that minimize the recognition loss caused by maximizing sparsity. This procedure zeros out more than 90% of parameters, with a drop of accuracy that is less than 1% on the ILSVRC2012 dataset. We also propose an efficient sparse matrix multiplication algorithm on CPU for Sparse Convolutional Neural Networks (SCNN) models. Our CPU implementation demonstrates much higher efficiency than the off-the-shelf sparse matrix libraries, with a significant speedup realized over the original dense network. In addition, we apply the SCNN model to the object detection problem, in conjunction with a cascade model and sparse fully connected layers, to achieve significant speedups.

We propose that the dynamics of an action in video data forms a sparse self-similar manifold in the space-time volume, which can be fully characterized by a linear rank decomposition. Inspired by the recurrence plot theory, we introduce the concept of Joint Self-Similarity Volume (Joint-SSV) to model this sparse action manifold, and hence propose a new optimized rank-1 tensor approximation of the Joint-SSV to obtain compact low-dimensional descriptors that very accurately characterize an action in a video sequence. We show that these descriptor vectors make it possible to recognize actions without explicitly aligning the videos in time in order to compensate for speed of execution or differences in video frame rates. Moreover, we show that the proposed method is generic, in the sense that it can be applied using different low-level features, such as silhouettes, tracked points, histogram of oriented gradients, and so forth. Therefore, our method does not necessarily require explicit tracking of features in the space-time volume. Our experimental results on five public data sets demonstrate that our method produces promising results and outperforms many baseline methods.

In this paper, we propose an unsupervised framework for action spotting in videos that does not depend on any specific feature (e.g. HOG/HOF, STIP, silhouette, bag-of-words, etc.). Furthermore, our solution requires no human localization, segmentation, or framewise tracking. This is achieved by treating the problem holistically as that of extracting the internal dynamics of video cuboids by modeling them in their natural form as multilinear tensors. To extract their internal dynamics, we devised a novel Two-Phase Decomposition (TP-Decomp) of a tensor that generates very compact and discriminative representations that are robust to even heavily perturbed data. Technically, a Rank-based Tensor Core Pyramid (Rank-TCP) descriptor is generated by combining multiple tensor cores under multiple ranks, allowing to represent video cuboids in a hierarchical tensor pyramid. The problem then reduces to a template matching problem, which is solved efficiently by using two boosting strategies: (1) to reduce search space, we filter the dense trajectory cloud extracted from the target video, (2) to boost the matching speed, we perform matching in an iterative coarse-to-fine manner. Experiments on 5 benchmarks show that our method outperforms current state-of-the-art under various challenging conditions. We also created a challenging dataset called Heavily Perturbed Video Array (HPVA) to validate the robustness of our framework under heavily perturbed situations.

Large-scale recognition problems with thousands of classes pose a particular challenge because applying the classifier requires more computation as the number of classes grows. The label tree model integrates classification with the traversal of the tree so that complexity grows logarithmically. In this paper, we show how the parameters of the label tree can be found using maximum likelihood estimation. This new probabilistic learning technique produces a label tree with significantly improved recognition accuracy.

While smoothness priors are ubiquitous in analysis of visual information, dictionary learning for image analysis has traditionally relied on local evidences only. We present a novel approach to discriminative dictionary learning with neighborhood constraints. This is achieved by embedding dictionaries in a Conditional Random Field (CRF) and imposing labeldependent smoothness constraints on the resulting sparse codes at adjacent sites. This way, a smoothness prior is used while learning the dictionaries and not just to make inference. This is in contrast with competing approaches that learn dictionaries without such a prior. Pixel-level classification results on the Graz02 bikes dataset demonstrate that dictionaries learned in our discriminative setting with neighborhood smoothness constraints can equal the state-of-the-art performance of bottom-up (i.e. superpixel-based) segmentation approaches. Furthermore, we isolate the benets of our learning formulation and CRF inference to show that our dictionaries are more discriminative than dictionaries learned without such constraints even without CRF inference. An additional benet of our smoothness constraints is more stable learning which is a known problem for discriminative dictionaries.

This brief proposes a continuously-valued Markov random field (MRF) model with separable filter bank, denoted as MRFSepa, which significantly reduces the computational complexity in the MRF modeling. In this framework, we design a novel gradient-based discriminative learning method to learn the potential functions and separable filter banks. We learn MRFSepa models with 2-D and 3-D separable filter banks for the applications of gray-scale/color image denoising and color image demosaicing. By implementing MRFSepa model on graphics processing unit, we achieve real-time image denoising and fast image demosaicing with high-quality results.

We describe how inflation, the act of mapping a 2D silhouette to a 3D region, can be applied in two disparate problems to offer insight and improvement: silhouette part segmentation and image-based material transfer. To demonstrate this, we introduce Puffball, a novel inflation technique, which achieves similar results to existing inflation approaches -- including smoothness, robustness, and scale and shift-invariance -- through an exceedingly simple and accessible formulation. The part segmentation algorithm avoids many of the pitfalls of previous approaches by finding part boundaries on a canonical 3-D shape rather than in the contour of the 2-D shape; the algorithm gives reliable and intuitive boundaries, even in cases where traditional approaches based on the 2D Minima Rule are misled. To demonstrate its effectiveness, we present data in which subjects prefer Puffball's segmentations to more traditional Minima Rule-based segmentations across several categories of silhouettes. The texture transfer algorithm utilizes Puffball's estimated shape information to produce visually pleasing and realistically synthesized surface textures with no explicit knowledge of either underlying shape.

In this paper, we design a novel MRF framework which is called Non-Local Range Markov Random Field (NLR-MRF). The local spatial range of clique in traditional MRF is extended to the non-local range which is defined over the local patch and also its similar patches in a non-local window. Then the traditional local spatial filter is extended to the non-local range filter that convolves an image over the non-local ranges of pixels. In this framework, we propose a gradient-based discriminative learning method to learn the potential functions and non-local range filter bank. As the gradients of loss function with respect to model parameters are explicitly computed, efficient gradient-based optimization methods are utilized to train the proposed model. We implement this framework for image denoising and in-painting, the results show that the learned NLR-MRF model significantly outperforms the traditional MRF models and produces state-of-the-art results.

This paper presents models and algorithms for estimating the shape of a mirrored surface from a single image of that surface, rendered under an unknown, natural illumination. While the unconstrained nature of this problem seems to make shape recovery impossible, the curvature of the surface cause characteristic image patterns to appear. These image patterns can be used to estimate how the surface curves in different directions. We show how these estimates can be used to produce constraints that can be used to estimate the shape of the surface. This approach is demonstrated on simple surfaces rendered under both natural and synthetic illuminations.

This paper proposes a context-constrained hallucination approach for image super-resolution. Through building a training set of high-resolution/low-resolution image segment pairs, the high-resolution pixel is hallucinated from its texturally similar segments which are retrieved from the training set by texture similarity. Given the discrete hallucinated examples, a continuous energy function is designed to enforce the fidelity of high-resolution image to low-resolution input and the constraints imposed by the hallucinated examples and the edge smoothness prior. The reconstructed high-resolution image is sharp with minimal artifacts both along the edges and in the textural regions.

For object category recognition to scale beyond a small number of classes, it is important that algorithms be able to learn from a small amount of labeled data per additional class. One-shot recognition aims to apply the knowledge gained from a set of categories with plentiful data to categories for which only a single exemplar is available for each. As with earlier efforts motivated by transfer learning, we seek an internal representation for the domain that generalizes across classes. However, in contrast to existing work, we formulate the problem in a fundamentally new manner by optimizing the internal representation for the one-shot task using the notion of micro-sets. A micro-set is a sample of data that contains only a single instance of each category, sampled from the pool of available data, which serves as a mechanism to force the learned representation to explicitly address the variability and noise inherent in the one-shot recognition task. We optimize our learned domain features so that they minimize an expected loss over micro-sets drawn from the training set and show that these features generalize effectively to previously unseen categories. We detail a discriminative approach for optimizing one-shot recognition using micro-sets and present experiments on the Animals with Attributes and Caltech-101 datasets that demonstrate the benefits of our formulation.

This paper addresses the problem of recognizing shadows from monochromatic natural images. Without chromatic information, shadow classification is very challenging because the invariant color cues are unavailable. Natural scenes make this problem even harder because of ambiguity from many near black objects. We propose to use both shadow-variant and shadow-invariant cues from illumination, textural and odd order derivative characteristics. Such features are used to train a classifier from boosting a decision tree and integrated into a Conditional random Field, which can enforce local consistency over pixel labels. The proposed approach is evaluated using both qualitative and quantitative results based on a novel database of hand-labeled shadows. Our results show shadowed areas of an image can be identified using proposed monochromatic cues.

We present a new approach for the discriminative training of continuous-valued Markov Random Field (MRF) model parameters. In our approach we train the MRF model by optimizing the parameters so that the minimum energy solution of the model is as similar as possible to the ground-truth. This leads to parameters which are directly optimized to increase the quality of the MAP estimates during inference. Our proposed technique allows us to develop a framework that is flexible and intuitively easy to understand and implement, which makes it an attractive alternative to learn the parameters of a continuous-valued MRF model. We demonstrate the effectiveness of our technique by applying it to the problems of image denoising and in-painting using the Field of Experts model. In our experiments, the performance of our system compares favourably to the Field of Experts model trained using contrastive divergence when applied to the denoising and in-painting tasks.

In this paper we describe a method to learn parameters which govern pedestrian motion by observing video data. Our learning framework is based on variational mode learning and allows us to efficiently optimize a continuous pedestrian cost model. We show that this model can be trained on automatic tracking results, and provides realistic and accurate pedestrian motions.

Abstract Incorrectly setting the camera's exposure can have a significant negative effect on a photograph. Over‐exposing photographs causes pixels to exhibit unpleasant artifacts due to saturation of the sensor. Saturation removal typically involves user intervention to adjust the color values, which is tedious and time‐consuming. This paper discusses how saturation can be automatically removed without compromising the essential details of the image. Our method is based on a smoothness prior: neighboring pixels have similar channel ratios and color values. We demonstrate that high quality saturation‐free photos can be obtained from a simple but effective approach.

Graphical models are fundamental tools for modeling images and other applications. In this paper, we propose the logistic random field (LRF) model for representing a discrete-valued graphical model. The LRF model is based on an underlying quadratic model and a logistic function. The chief advantages of the LRF are its convenience and flexibility. The quadratic model makes inference easy to implement using standard numerical linear algebra routines. This quadratic model also allows the log-likelihood of the training data to be differentiated with respect to any parameter in the model, enhancing the flexibility of the LRF model. To demonstrate the usefulness of this model we use it to learn how to segment objects, specifically roads, horses, and cows. In addition, we demonstrate the flexibility of the LRF model by incorporating super-pixels. We then show that the LRF segmentation model produces segmentations that are competitive with recently published results.

Among the most exciting advances in early vision has been the development of efficient energy minimization algorithms for pixel-labeling tasks such as depth or texture computation. It has been known for decades that such problems can be elegantly expressed as Markov random fields, yet the resulting energy minimization problems have been widely viewed as intractable. Recently, algorithms such as graph cuts and loopy belief propagation (LBP) have proven to be very powerful: for example, such methods form the basis for almost all the top-performing stereo methods. However, the tradeoffs among different energy minimization algorithms are still not well understood. In this paper we describe a set of energy minimization benchmarks and use them to compare the solution quality and running time of several common energy minimization algorithms. We investigate three promising recent methods graph cuts, LBP, and tree-reweighted message passing in addition to the well-known older iterated conditional modes (ICM) algorithm. Our benchmark problems are drawn from published energy functions used for stereo, image stitching, interactive segmentation, and denoising. We also provide a general-purpose software interface that allows vision researchers to easily switch between optimization methods. Benchmarks, code, images, and results are available at http://vision.middlebury.edu/MRF/.

Markov random field, or MRF, models are a powerful tool for modeling images. While much progress has been made in algorithms for inference in MRFs, learning the parameters of an MRF is still a challenging problem. In this paper, we show how variational optimization can be used to learn the parameters of an MRF. This method for learning, which we refer to as variational mode learning, finds the MRF parameters by minimizing a loss function that penalizes the difference between ground-truth images and an approximate, variational solution to the MRF. In particular, we focus on learning parameters for the field of experts model of Roth and Black. In addition to demonstrating the effectiveness of this method, we show that a model based on derivative filters performs quite similarly to the field of experts model. This suggests that the field of experts model, which is difficult to interpret, can be understood as imposing piecewise continuity on the image.

Markov random field (MRF) models are a popular tool for vision and image processing. Gaussian MRF models are particularly convenient to work with because they can be implemented using matrix and linear algebra routines. However, recent research has focused on on discrete-valued and non-convex MRF models because Gaussian models tend to over-smooth images and blur edges. In this paper, we show how to train a Gaussian conditional random field (GCRF) model that overcomes this weakness and can outperform the non-convex field of experts model on the task of denoising images. A key advantage of the GCRF model is that the parameters of the model can be optimized efficiently on relatively large images. The competitive performance of the GCRF model and the ease of optimizing its parameters make the GCRF model an attractive option for vision and image processing applications.

Images can be represented as the composition of multiple intrinsic component images, such as shading, albedo, and noise images. In this paper, we present a method for estimating intrinsic component images from a single image, which we apply to the problems of estimating shading and albedo images and image denoising. Our method is based on learning estimators that predict filtered versions of the desired image. Unlike previous approaches, our method does not require unnatural discretizations of the problem. We also demonstrate how to learn a weighting function that properly weights the local estimates when constructing the estimated image. For shading estimation, we introduce a new training set of real-world images. The accuracy of our method is measured both qualitatively and quantitatively, showing better performance on the shading/albedo separation problem than previous approaches. The performance on denoising is competitive with the current state of the art.

Interpreting real-world images requires the ability distinguish the different characteristics of the scene that lead to its final appearance. Two of the most important of these characteristics are the shading and reflectance of each point in the scene. We present an algorithm that uses multiple cues to recover shading and reflectance intrinsic images from a single image. Using both color information and a classifier trained to recognize gray-scale patterns, given the lighting direction, each image derivative is classified as being caused by shading or a change in the surface's reflectance. The classifiers gather local evidence about the surface's form and color, which is then propagated using the generalized belief propagation algorithm. The propagation step disambiguates areas of the image where the correct classification is not clear from local evidence. We use real-world images to demonstrate results and show how each component of the system affects the results.

Recent stereo algorithms have achieved impressive results by modelling the disparity image as a Markov Random Field (MRF). An important component of an MRF-based approach is the inference algorithm used to find the most likely setting of each node in the MRF. Algorithms have been proposed which use graph cuts or belief propagation for inference. These stereo algorithms differ in both the inference algorithm used and the formulation of the MRF. It is unknown whether to attribute the responsibility for differences in performance to the MRF or the inference algorithm. We address this through controlled experiments by comparing the belief propagation algorithm and the graph cuts algorithm on the same MRF's, which have been created for calculating stereo disparities. We find that the labellings produced by the two algorithms are comparable. The solutions produced by graph cuts have a lower energy than those produced with belief propagation, but this does not necessarily lead to increased performance relative to the ground truth.

We present an algorithm that uses multiple cues to recover shading and reflectance intrinsic images from a single image. Using both color in- formation and a classifier trained to recognize gray-scale patterns, each image derivative is classified as being caused by shading or a change in the surface’s reflectance. Generalized Belief Propagation is then used to propagate information from areas where the correct classification is clear to areas where it is ambiguous. We also show results on real images.