In ground-based remote sensing cloud image observation, images with the highest possible resolution are captured to obtain sufficient information about clouds. However, when features are extracted and classification is performed on the basis of the original images, a high-resolution probably means a high (or even more, unacceptable) computation cost. In practical application, a simple and commonly adopted method is to appropriately resize the original image to a version with a decreased resolution. An inevitable problem is whether useful information is lost in this resizing operation. This paper demonstrates that information loss is inevitable and poor classification results may be obtained from the analysis of local binary pattern (LBP) histogram features. However, this problem has been always neglected in previous studies, and the original image is arbitrarily resized without any criterion. In particular, the histogram features based on LBPs actually reflect the distribution of features. Thus, a criterion based on the Kullback-Leibler divergence between LBP histograms from the original and resized images and a penalty term imposed on the resolution are proposed to select the resolution of the resized image. The optimal resolution of the resized image can be selected by minimizing this criterion. Furthermore, experiments based on three ground-based remote sensing cloud image data sets with different original resolutions validate this criterion by analyzing the LBP histogram features.

In this paper, we propose a simple but effective semantic-based aggregation (SBA) method. The proposed SBA utilizes the discriminative filters of deep convolutional layers as semantic detectors. Moreover, we propose the effective unsupervised strategy to select some semantic detectors to generate the "soft region proposals," which highlight certain discriminative pattern of objects and suppress the noise of background. The final global SBA representation could then be acquired by aggregating the regional representations weighted by the selected "soft region proposals" corresponding to various semantic content. Our unsupervised SBA is easy to generalize and achieves excellent performance on various tasks. We conduct comprehensive experiments and show that our unsupervised SBA outperforms the state-of-the-art unsupervised and supervised aggregation methods on image retrieval, place recognition, and cloud classification.

Existing manifold learning methods are not appropriate for image retrieval tasks, because most of them are unable to process query images and they have much greater computational cost especially for large-scale database. Therefore, we propose the iterative manifold embedding (IME) layer, of which the weights are learned offline by an unsupervised strategy, to explore the intrinsic manifolds by incomplete data. On the large-scale database that contains 27 000 images, the IME layer is more than 120 times faster than other manifold learning methods to embed the original representations at query time. We embed the original descriptors of database images that lie on manifold in a high-dimensional space into manifold-based representations iteratively to generate the IME representations in an offline learning stage. According to the original descriptors and the IME representations of database images, we estimate the weights of the IME layer by ridge regression. In the online retrieval stage, we employ the IME layer to map the original representation of a query image with an ignorable time cost (2 ms per image). We experiment on five public standard datasets for image retrieval. The proposed IME layer significantly outperforms the related dimension reduction methods and manifold learning methods. Without postprocessing, our IME layer achieves a boost in the performance of state-of-the-art image retrieval methods with postprocessing on most datasets, and needs less computational cost. The code is available at https://github.com/XJhaoren/IME_layer.

In this paper, we propose a simple but effective semantic part-based weighting aggregation (PWA) for image retrieval. The proposed PWA utilizes the discriminative filters of deep convolutional layers as part detectors. Moreover, we propose the effective unsupervised strategy to select some part detectors to generate the "probabilistic proposals," which highlight certain discriminative parts of objects and suppress the noise of background. The final global PWA representation could then be acquired by aggregating the regional representations weighted by the selected "probabilistic proposals" corresponding to various semantic content. We conduct comprehensive experiments on four standard datasets and show that our unsupervised PWA outperforms the state-of-the-art unsupervised and supervised aggregation methods.

Many researchers make numerous work on document image binarization. However, the binarization results of camera-captured document images remain to be improved due to many disturbances such as creases, noises and shadows. To binarize these images effectively, this paper proposes an adaptive local thresholding method which takes advantages of multi-level multi-scale local statistical information. By using the context information of multiple scales, the pixels in the image are classified by coarse to fine. The majority of background areas were removed by multiscale analysis of variance. For the text area, the binarization threshold is dynamically adjusted according to the estimated clarity. Our method can make the grayscale image binarization directly, without adding any postprocessing operation. The experimental results show that our method can significantly improve the performance of OCR system and is also suitable for degraded document images.

Cloud type classification based on ground-based cloud image observations is an important task in atmospheric research. Currently, two kinds of cloud image observations with infrared and visible light images are widely used for cloud classification. However, they are only independently analyzed and simply compared in the current study. The useful information from these two kinds of images is not fully utilized and integrated. The classification performance could be improved if taking full advantage of the complementary information of these two observations. Thus, first, a database containing these two kinds of cloud images with same temporal resolution is released in this study. Then, a two-observation joint encoding strategy of LBP (local binary pattern) features is proposed to implement cloud classification by encoding the joint distribution of LBP patterns in different observations, which captures the correlation between two observations. Experimental results based on this database show the significant superiority of the proposed method compared to the results based on the single observation.

The faint text document images possess shallow characters inherently and the camera-captured form introduces more degradations such as low-resolution, non-uniform illumination and out-of-focus blur, which make the text binarization very difficult. In this paper, we propose a grayscale-projection based optimal character segmentation method for camera-captured faint text recognition. Instead of extracting the character candidates, we use the gradient projection to extract a series of segmentation candidates which contain inter-character gaps and intra-character gaps as well. In order to select the optimal segmentation path from all possible situations, we construct a segmentation tree and set a evaluation score for each path. The score integrates the information of single point projection, overall distribution and recognition probability. Finally the optimal segmentation path is obtained by selecting the path with the highest score. We collect a faint text recognition dataset and evaluate our method on it. Experimental results show that our method outperforms the binary-projection method and the convolutional recurrent neural network approach in terms of text segmentation and recognition accuracy.

Recognizing scene character is extremely challenging due to various interference factors such as character translation, blur and uneven illumination, etc. Considering that characters are composed of a series of parts and different parts attract diverse attentions when people observe a character, we should assign different importance to each part to recognize scene character. In this paper, we propose a discriminative character representation by aggregating the responses of the spatially embedded salient part detectors. Specifically, we first extract the convolution activations from the pre-trained convolutional neural network (CNN). These convolutional activations are considered as the local descriptors of the character parts. Then we learn a set of part detectors and pick the distinctive convolutional activations which respond to the salient parts. Moreover, to alleviate the effect of character translation, rotation and deformation, etc, we assign a response region for each part detector and search the maximal response in this region. Finally, we aggregate the maximal outputs of all the salient part detectors to represent character. The experiments on three datasets show the effectiveness of the proposed method for scene character recognition.

Several recent works interpret convolutional features produced by deep convolutional neural networks as local descriptors. Existing high-dimensional aggregation based methods, e.g., Fisher Vector (FV) obtain inferior performance to pooling based methods in most situations, and we observe that it is mainly caused by the ignorance of spatial weights. In this paper, we propose a novel method named spatial weighted Fisher Vector (SWFV) to enhance the representation of FV by injecting the spatial weight map to FV. In addition, we further analyze the distribution of spatial weights and propose truncated spatial weighted FV (TSWFV). Experimental results on two benchmark datasets demonstrate that the two proposed methods achieve competitive results compared with other global representation based methods.

Cloud detection plays an important role in climate models, climate predictions, and meteorological services. Although researchers have given increasing efforts on cloud detection, the performance is still unsatisfactory due to the diverse nature of clouds. Considering the fact that one source of information (color or texture) is not enough to segment cloud from clear sky, in this letter, we propose a novel ground-based cloud detection method using graph model (GM) built upon superpixels to integrate multiple sources of information. First, we use the superpixel segmentation to divide the image into a series of subregions according to the color similarity and spatial continuity. Next, adjacent superpixels are merged according to their similarity of extracted features. Finally, we build a GM on the merged superpixels by considering each superpixel as a node and adding edges between neighboring ones. The unary cost is set according to the classification score of Random Forests, while pairwise cost reflects the penalties for color and texture discontinuity between neighboring components. The final segmentation could be acquired by minimizing the cost function. Moreover, the algorithm is computationally efficient as we use the superpixels rather than raw pixels as computation units. Experimental results demonstrate the effectiveness and efficiency of the proposed method for cloud detection.

Ground-based cloud classification is crucial for meteorological research and has received great concern in recent years. However, it is very challenging due to the extreme appearance variations under different atmospheric conditions. Although the convolutional neural networks have achieved remarkable performance in image classification, no one has evaluated their suitability for cloud classification. In this letter, we propose to use the deep convolutional activations-based features (DCAFs) for ground-based cloud classification. Considering the unique characteristic of cloud, we believe the local rich texture information might be more important than the global layout information and, thus, give a comprehensive evaluation of using both shallow convolutional layers-based features and DCAFs. Experimental results on two challenging public data sets demonstrate that although the realization of DCAF is quite straightforward without any use-dependent tricks, it outperforms conventional hand-crafted features considerably.

Currently, ground-based cloud images taken by using a whole-sky imager are especially popular in the field of meteorology because of their high resolution and accurate cloud information. Cloud images are natural texture images, and thus texture features based on local binary patterns (LBPs) are widely used to analyze texture images. However, the high-computation cost of extracting LBP features from high-resolution cloud texture images may make this technique unacceptable in practical image processing. A commonly adopted method is to resize the original image to an appropriate version with a decreased resolution. But this process will inevitably result in information loss. Accordingly, a measure based on the Kullback-Leibler (KL) divergence of the difference between LBP histogram features extracted from the original and resized images with varying resolutions is reported in this letter. Furthermore, a confidence interval technique is introduced to validate the significance of such difference. Experiments based on real ground-based cloud images show the measurement results of KL divergence in LBP features extracted from original and resized images. The experimental results indicate that images should be resized with caution when performing image processing.

This letter presents a novel approach for logo retrieval. Considering the fact that logo only occupies a small portion of an image, we apply Faster R-CNN to detect logo proposals first, and then use a two-step pooling strategy with adaptive weight to obtain an accurate global signature. The adaptive weighted pooling method can effectively balance the recall and precision of proposals by incorporating the probability of each proposal being a logo. Experimental results show that the proposed method interprets the similarity between query and database image more accurately and achieves state of the art performance.

In this paper, a novel local threshold binarization method using structural symmetry of strokes is proposed. Different from most existing local threshold methods which use the whole region to compute the threshold, we estimate the local threshold by only using the structural symmetric pixels (SSP) of the region so as to suppress the non-text pixels and maintain the text ones as well. The SSP is defined as those pixels around strokes whose gradient magnitudes are big enough and directions are opposite. As the gradient map is our basis for computing the SSP, we further propose to estimate background surface first and extract potential SSP in the compensated image so as to deal with degradations of document images such as uneven illumination, low contrast and stain. To prove the effectiveness of our method, tests on two public document image datasets are preformed and the experimental results show that our method outperforms other local threshold binarization approaches on both F-measure and PSNR.

Gamma correction, a nonlinear operation, has long been used to code and decode luminance or tristimulus values in video or still image systems [1]. In this paper, we make the following observations: for CAPTCHA images which could not be well binarized using the threshold of OTSU, there exists a gamma corrected image which could be well segmented by the OTSU threshold and the value of the best gamma could be revealed by observing the maximal inter-class variance (MICV) values of different images transformed by different values of gamma. Concretely, we convert the R, G, B channels of the original CAPTCHA image with different gamma values and transform the color images to gray-level images. Each gray-level image could be then segmented by the threshold acquired by OTSU. By linking each gamma value with the corresponding maximal inter-class variance value, we could draw a changing curve of variance values versus gamma. The best gamma could be acquired by finding the point whose related MICV starts to change slowly. Moreover, the polarity of the image could also be revealed by the changing trend of the curve. Experimental results on different categories of CAPTCHA images demonstrate the effectiveness of the observations for binarizing the CAPTCHA images and telling the polarity as well.

This paper presents a novel binarization technique for text images based on Markov Random Field (MRF) framework. We regard stroke as an obvious feature of text to produce clustering result, which will be optimized by MRF model combining color, texture, context features to get the final binarization. The main innovations of our method are: (1) the integrated image is split into sub-images on which we can automatically acquire seed pixels of foreground and background using stroke feature; and (2) diverse weights are attached to seed pixels according to their location information, then highly confident cluster centers of sub-image can be acquired by gathering weighted seeds. The experimental results show that our method is robust and accurate on both video and scene images.

Text information contained in scene images is very helpful for high‐level image understanding. In this study, the authors propose to learn co‐occurrence of local strokes for scene text recognition by using a spatiality embedded dictionary (SED). Unlike spatial pyramid partitioning images into grids to incorporate spatial information, the authors SED associates every codeword with a particular response region and introduces more precise spatial information for robust character recognition. After localised soft coding and max pooling of the first layer, a sparse dictionary is learned to model co‐occurrence of several local strokes, which further improves classification performance. Experimental results on two scene character recognition datasets ICDAR2003 and CHARS74 K demonstrate that their character recognition method outperforms state‐of‐the‐art methods. Besides, competitive word recognition results are also reported for four benchmark word recognition datasets ICDAR2003, ICDAR2011, ICDAR2013 and street view text when combining their character recognition method with a conditional random field language model.

Characters, which are man-made symbols composed of strokes arranged in a certain structure, could provide semantic information and play an indispensable role in our daily life. In this paper, we try to make use of the intrinsic characteristics of characters and explore the stroke and structure-based methods for character recognition. First, we introduce two existing part-based models to recognize characters by detecting the elastic strokelike parts. In order to utilize strokes of various scales, we propose to learn the discriminative multi-scale stroke detector-based representation (DMSDR) for characters. However, the part-based models and DMSDR need to manually label the parts or key points for training. In order to learn the discriminative stroke detectors automatically, we further propose the discriminative spatiality embedded dictionary learning-based representation (DSEDR) for character recognition. We make a comparative study of the performance of the tree-structured model (TSM), mixtures-of-parts TSM, DMSDR, and DSEDR for character recognition on three challenging scene character recognition (SCR) data sets as well as two handwritten digits recognition data sets. A series of experiments is done on these data sets with various experimental setup. The experimental results demonstrate the suitability of stroke detector-based models for recognizing characters with deformations and distortions, especially in the case of limited training samples.

Robust scene-text-extraction system can be used in lots of areas. In this work, we propose to learn co-occurrence of local strokes for robust character recognition by using a spatiality embedded dictionary (SED). Different from spatial pyramid partitioning images into grids to incorporate spatial information, our SED associates every codeword with a particular response region and introduces more precise spatial information for character recognition. After localized soft coding and max pooling of the first layer, a sparse dictionary is learned to model co-occurrence of several local strokes, which further improves classification performance. Experiment on benchmark datasets demonstrates the effectiveness of our method and the results outperform state-of-the-art algorithms.

Text information contained in scene images is very useful for image understanding. In this paper, we propose a high-level representation named stroke bank for scene character recognition. Inspired by the work of object bank, we train stroke detectors and use detectors' maximal output as features. Specifically, we collect training samples for stroke detectors based on labeled key points. We also propose to restrict classification areas of each stroke detector to particular local regions, which alleviates computation burden and retains discrimination power at the same time. Experiments on benchmark datasets demonstrate the effectiveness of our method and the results outperform state-of-the-art algorithms.

Scene text recognition has inspired great interests from the computer vision community in recent years. In this paper, we propose a novel scene text-recognition method integrating structure-guided character detection and linguistic knowledge. We use part-based tree structure to model each category of characters so as to detect and recognize characters simultaneously. Since the character models make use of both the local appearance and global structure informations, the detection results are more reliable. For word recognition, we combine the detection scores and language model into the posterior probability of character sequence from the Bayesian decision view. The final word-recognition result is obtained by maximizing the character sequence posterior probability via Viterbi algorithm. Experimental results on a range of challenging public data sets (ICDAR 2003, ICDAR 2011, SVT) demonstrate that the proposed method achieves state-of-the-art performance both for character detection and word recognition.

Scene text recognition has inspired great interests from the computer vision community in recent years. In this paper, we propose a novel scene text recognition method using part-based tree-structured character detection. Different from conventional multi-scale sliding window character detection strategy, which does not make use of the character-specific structure information, we use part-based tree-structure to model each type of character so as to detect and recognize the characters at the same time. While for word recognition, we build a Conditional Random Field model on the potential character locations to incorporate the detection scores, spatial constraints and linguistic knowledge into one framework. The final word recognition result is obtained by minimizing the cost function defined on the random field. Experimental results on a range of challenging public datasets (ICDAR 2003, ICDAR 2011, SVT) demonstrate that the proposed method outperforms state-of-the-art methods significantly both for character detection and word recognition.

In this paper, we propose a novel method for cross-view action recognition via a continuous virtual path which connects the source view and the target view. Each point on this virtual path is a virtual view which is obtained by a linear transformation of the action descriptor. All the virtual views are concatenated into an infinite-dimensional feature to characterize continuous changes from the source to the target view. However, these infinite-dimensional features cannot be used directly. Thus, we propose a virtual view kernel to compute the value of similarity between two infinite-dimensional features, which can be readily used to construct any kernelized classifiers. In addition, there are a lot of unlabeled samples from the target view, which can be utilized to improve the performance of classifiers. Thus, we present a constraint strategy to explore the information contained in the unlabeled samples. The rationality behind the constraint is that any action video belongs to only one class. Our method is verified on the IXMAS dataset, and the experimental results demonstrate that our method achieves better performance than the state-of-the-art methods.

Detecting text in scene images is very challenging due to complex backgrounds, various fonts and different illumination conditions. Without prior knowledge, a detector previously trained using lots of samples still perform badly on a test image because of the disparities in distributions between the training samples and the testing ones. In this paper, we propose to adapt a pre-trained generic scene text detector towards new scenes by transfer learning. In particular, we choose cascade Adaboost as the detector style and try to re-weight pre-selected features according to their abilities to classify high confidence samples. The proposed adaptation mechanism has been evaluated on ICDAR 2011 scene text detection competition dataset and the encouraging experiments results can be compared with the latest published algorithms.

Interactive image segmentation which needs the user to give certain hard constraints has shown promising performance for object segmentation. In this paper, we consider characters in text image as a special kind of object, and propose an adaptive graph cut based text binarization method to segment text from background. The main contributions of the paper lie in: 1) in order to make the binarization local adaptive with uneven background, the text region image is firstly roughly split into several sub-images on which graph cut is applied, and 2) considering the unique characteristics of the text, we propose to automatically classify some pixels as text or background with high confidence, severed as hard constraints seeds for graph cut to extract text from background by spreading the seeds into the whole sub-image. The experimental results show that our approach could get better performance in both character extraction accuracy and recognition accuracy.

Detecting text in video or natural scene image is quite challenging due to the complex background, various fonts and illumination conditions. The preprocessing period, which suppresses the nontext areas so as to highlight the text areas, is the basis for further text detection. In this paper, a novel graph-based background suppression method for scene text detection is proposed. Considering each pixel as a node in the graph, our approach incorporates pixel-level and context-level features into a graph. Various factors contribute to the unary and pair wise cost function which is optimized via max-flow/min-cut algorithm [16] to get a binary image whose nontext pixels are suppressed so that text pixels are highlighted. Furthermore, the proposed background suppression method could be easily combined with other detection methods to improve the performance. Experimental results on ICDAR 2011 competition dataset show promising performance.

Text in image or video frames contains a lot of high-level semantics which can be useful for multimedia indexing, management. Coarse text detection results may contain many false alarms, which makes it necessary to eliminate the false alarms for further recognition. As text has distinct textural features, texture-based classifier such as SVM, MLP and Adaboost has been used to classify the detection regions as text or non-text region. In this paper, a random forests based method for text verification is proposed. The reason of choosing random forests lies in: 1) its ability of maintaining accuracy in small labeled dataset and 2) its good performance in unbalanced dataset as in the case of unbalanced text and non-text distribution. Furthermore, we propose to merge different random forests trained with different kinds of features to improve the accuracy of classification. The comprehensive experimental results show that our methods are effective.

Text detection and extraction in images with complex background can provide useful information for video annotation and indexing. More attention is paid to text detection for its importance, but text extraction is necessary for the text recognition, and it can test the validity of text detection. In this paper, we conclude text extraction is to segment the image and to remove noises, and then a robust text extraction method incorporating local information is proposed. First, we get the gray image from the original image and reprocess the gray image with edge enhancement. Then a binarization method incorporating local information is used to segment the gray image, by which the text-noises are removed and a binary image is obtained. Finally, the connected component analysis based on the character's density and geometric feature is performed on the binary image, by which background-noises are removed. The preliminary experiments show some promising results.