In this paper, we propose a multi-task learning approach for table detection and cell segmentation with densely connected graph attention networks in free form online documents. Each online document is regarded as a graph, where nodes represent strokes and edges represent the relationships between strokes. Then we propose a graph attention network model to classify nodes and edges simultaneously. According to node classification results, tables can be detected in each document. By combining node and edge classification resutls, cells in each table can be segmented. To improve information flow in the network and enable efficient reuse of features among layers, dense connectivity among layers is used. Our proposed model has been experimentally validated on an online handwritten document dataset IAMOnDo and achieved encouraging results.

In contrast to machine recognizers that rely on training with large handwriting data, humans can recognize handwriting accurately on learning from few samples, and can even generalize to handwritten characters from printed samples. Simulating this ability in machine recognition is important to alleviate the burden of labeling large handwriting data, especially for large category set as in Chinese text. In this paper, inspired by human learning, we propose a cross-modal prototype learning (CMPL) method for zero-shot online handwritten character recognition: for unseen categories, handwritten characters can be recognized without learning from handwritten samples, but instead from printed characters. Particularly, the printed characters (one for each class) are embedded into a convolutional neural network (CNN) feature space to obtain prototypes representing each class, while the online handwriting trajectories are embedded with a recurrent neural network (RNN). Via cross-modal joint learning, handwritten characters can be recognized according to the printed prototypes. For unseen categories, handwritten characters can be recognized by only feeding a printed sample per category. Experiments on a benchmark Chinese handwriting database have shown the effectiveness and potential of the proposed method for zero-shot handwriting recognition.

Segmentation of complex document images remains a challenge due to the large variability of layout and image degradation. In this paper, we propose a method to segment complex document images based on Label Pyramid Network (LPN) and Deep Watershed Transform (DWT). The method can segment document images into instance aware regions including text lines, text regions, figures, tables, etc. The backbone of LPN can be any type of Fully Convolutional Networks (FCN), and in training, label map pyramids on training images are provided to exploit the hierarchical boundary information of regions efficiently through multi-task learning. The label map pyramid is transformed from region class label map by distance transformation and multi-level thresholding. In segmentation, the outputs of multiple tasks of LPN are summed into one single probability map, on which watershed transformation is carried out to segment the document image into instance aware regions. In experiments on four public databases, our method is demonstrated effective and superior, yielding state of the art performance for text line segmentation, baseline detection and region segmentation.

Data hiding is an effective technique, compared to pervasive black-and-white code patterns such as barcode and quick response code, which can be used to secure document images against forgery or unauthorized intervention. In this work, we propose a robust digital watermarking scheme for securing genuine documents by leveraging generative adversarial networks (GAN). To begin with, the input document is adjusted to its right form by geometric correction. Next, the generated document is obtained from the input document by using the mentioned networks, and it is regarded as a reference for data hiding and detection. We then introduce an algorithm that hides a secret information into the document and produces a watermarked document whose content is minimally distorted in terms of normal observation. Furthermore, we also present a method that detects the hidden data from the watermarked document by measuring the distance of pixel values between the generated and watermarked document. For improving the security feature, we encode the secret information prior to hiding it by using pseudo random numbers. Lastly, we demonstrate that our approach gives high precision of data detection, and competitive performance compared to state-of-the-art approaches.

Motivated by increasing possibility of the tampering of genuine documents during a transmission over digital channels, we focus on developing a watermarking framework for determining whether a given document is genuine or falsified. The proposed framework is performed by hiding a security feature or secret information within the document. In order to hide the security feature, we replace the appropriate characters of legal document by the equivalent characters coming from generated fonts, called hereafter the variations of characters. These variations are produced by training generative adversarial networks (GAN) with the features of character's skeleton and normal shape. Regarding the process of detecting hidden information, we make use of fully convolutional networks (FCN) to produce salient regions from the watermarked document. The salient regions mark positions of document where the characters are substituted by their variations, and these positions are used as a reference for extracting the hidden information. Lastly, we demonstrate that our approach gives high precision of data detection, and competitive performance compared to state-of-the-art approaches.

With the growing cosmopolitan culture of modern cities, the need of robust Multi-Lingual scene Text (MLT) detection and recognition systems has never been more immense. With the goal to systematically benchmark and push the state-of-the-art forward, the proposed competition builds on top of the RRC-MLT-2017 with an additional end-to-end task, an additional language in the real images dataset, a large scale multi-lingual synthetic dataset to assist the training, and a baseline End-to-End recognition method. The real dataset consists of 20,000 images containing text from 10 languages. The challenge has 4 tasks covering various aspects of multi-lingual scene text: (a) text detection, (b) cropped word script classification, (c) joint text detection and script classification and (d) end-to-end detection and recognition. In total, the competition received 60 submissions from the research and industrial communities. This paper presents the dataset, the tasks and the findings of the presented RRC-MLT-2019 challenge.

This paper introduces a Chinese Ancient Handwritten Characters Database (CASIA-AHCDB) for character recognition research. The database was built by annotating 11,937 pages of Chinese ancient handwritten documents. It consists of more than 2.2 million annotated handwritten character samples of 10,350 categories. According to the source of these documents, the database is divided into two datasets of different styles: Complete Library in Four Sections (AHCDB-style1) and Ancient Buddhist Scriptures (AHCDB-style2). Each dataset can be divided into three parts based on its applications. The first part, called basic category set, contains samples of common categories in two datasets, and is suitable for basic character recognition task. The second part, called enhanced category set, is mainly used for open-set character recognition task based on the basic character recognition. The third part, called the reserved category set, can be used in many pattern recognition tasks in the future. Based on the large category set, the various writing styles and the imbalanced sample number per category, CASIA-AHCDB can also be used for various classification and learning tasks such as transfer learning, few-shot learning. We performed experiments of basic character recognition on the basic category set, and report the results for benchmark. More techniques can be evaluated on this challenging database in the future.

Classifying strokes into different categories is an essential preprocessing step in the automatic document understanding process. To tackle this task, it is crucial to integrate different types of contextual information. Previous methods which are based on conditional random fields or recurrent neural networks have some limitations in model capacity or computational cost. In this paper, we propose a novel framework based on graph attention networks to solve this problem, which casts the stroke classification problem into the node classification problem in a document graph. In the graph, each node represents a stroke and the edges are built from temporal and spatial interactions between strokes. Combined graph convolution with attention mechanisms to dynamically aggregate features from the neighborhood, our model is very flexible to control the message passing routine between different nodes and therefore has strong capability learning context-aware features. We perform comparison experiments on the IAMonDo dataset and experimental results demonstrate the superiority of our approach.

Oracle character is one kind of the earliest hieroglyphics, which can be dated back to Shang Dynasty in China. Oracle character recognition is important for modern archaeology, ancient text understanding, and historical chronology, etc. To overcome the limitation and class imbalance of training data in oracle character recognition, we propose a classification method based on deep metric learning. We use a convolutional neural network (CNN) to map the character images to an Euclidean space where the distance between different samples can measure their similarities such that classification can be performed by the Nearest Neighbor (NN) rule. Because new categories are still being discovered in reality, our model enables the rejection of unseen categories and the configuration of new categories. To accelerate NN classification, we also propose a prototype pruning method with little loss of accuracy. The proposed method exceeds the state of the art on the public dataset Oracle-20K and outperforms CNN with softmax layer on a new dataset Oracle-AYNU.

In this paper we propose a structured prediction based system for text/non-text classification and printed/handwritten texts separation at connected component (CC) level in complex documents. We formulate the separation of different elements as joint classification problems and use conditional random fields (CRFs) to integrate both local and contextual information for improving the classification accuracy. Both our unary and pairwise potentials are formulated as neural networks for better exploiting contextual information. Considering the different properties in text/non-text classification and printed/handwritten texts separation, we use multilayer perception (MLP) and convolutional neural network (CNN) for potentials, respectively. To evaluate the performance of the proposed method, we provide a test paper document database named TestPaper1.0, which can be used for many other tasks as well. Our method achieve impressive results for both tasks on TestPaper1.0 dataset. Moreover, even with very shallow CNNs as potentials, our method achieves state-of-the-art performance for writing type (printed/handwritten) separation on the highly heterogeneous Maurdor dataset, surpassing Maurdor2013 and Maurdor2014 campaign winners. This demonstrates the effectiveness and superiority of our method.

Online video text detection is important in many applications, such as real-time translator and wearable camera system for visually-impaired. Existing methods for video text detection perform unsatisfactorily mainly because of the inferior text detection result and insufficient utilization of spatial and temporal information. Besides, the majority of them work in offline mode. In this paper, we propose an online video text detection method which works nearly in real time. We detect texts in each frame using a EAST based text detector, and formulate the online text tracking problem as decision making in Markov Decision Processes (MDPs). The similarity function in tracking stage can be learned by reinforcement learning. Besides, text detection and tracking are naturally unified by state transactions in the MDP. Extensive experiments on three benchmark datasets, ICDAR 2015, Minetto, and Youtube Video Text, verify the effectiveness of our method.

In this paper, we propose a method for simultaneous script identification and handwritten text line recognition in multi-task learning framework. Firstly, we use Separable Multi-Dimensional Long Short-Term Memory (SepMDLSTM) to encode the input text line images based on convolutional feature extraction. Then, the extracted features are fed into two classification modules for script identification and multi-script text recognition, respectively. All the network parameters are trained end-to-end by multi-task learning where the script identification task and the text recognition task are aimed to minimize the Negative Log Likelihood (NLL) loss and Connectionist Temporal Classification (CTC) loss, respectively. We evaluated the performance of the proposed method on handwritten text line datasets of three languages, namely, IAM (English), Rimes (French) and IFN/ENIT (Arabic). Experimental results demonstrate the multi-task learning framework performs superiorly for both script identification and text recognition. Particularly, the accuracy of script identification is higher than 99.9% and the character error rate (CER) of text recognition is even lower than that of some single-script text recognition systems.

Text detection and recognition in a natural environment are key components of many applications, ranging from business card digitization to shop indexation in a street. This competition aims at assessing the ability of state-of-the-art methods to detect Multi-Lingual Text (MLT) in scene images, such as in contents gathered from the Internet media and in modern cities where multiple cultures live and communicate together. This competition is an extension of the Robust Reading Competition (RRC) which has been held since 2003 both in ICDAR and in an online context. The proposed competition is presented as a new challenge of the RRC. The dataset built for this challenge largely extends the previous RRC editions in many aspects: the multi-lingual text, the size of the dataset, the multi-oriented text, the wide variety of scenes. The dataset is comprised of 18,000 images which contain text belonging to 9 languages. The challenge is comprised of three tasks related to text detection and script classification. We have received a total of 16 participations from the research and industrial communities. This paper presents the dataset, the tasks and the findings of this RRC-MLT challenge.

The digitization of Chinese historical documents poses a new challenge that in the huge set of character categories, majority of characters are not in common use now and have few samples for training the character classifiers. To settle this problem, we consider the radical-level composition of Chinese characters, and propose to detect position-dependent radicals using a deep residual network with multi-labeled learning. This enables the recognition of novel characters without training samples if the characters are composed of radicals appearing in training samples. In multi-labeled learning, each training character sample is labeled as positive for each radical it contains, such that after training, all the radicals appearing in the character can be detected. Experimental results on a large-category-set database of printed Chinese characters demonstrate that the proposed method can detect radicals accurately. Moreover, according to radical configurations, our model can credibly recognize novel characters as well as trained characters.

Maximally stable extremal region (MSER) is popularly used for candidate character candidate extraction in scene text detection. Its requirement of maximum stability hinders high performance on images of high variability. In this paper, we propose a novel character candidate extraction method based on superpixel segmentation and hierarchical clustering. The proposed superpixel segmentation algorithm for scene text image takes advantage of the color consistency of characters and fuses color and edge information. Based on superpixel segmentation, character candidates are extracted by single-link clustering. To improve the accuracy of non-text candidate filtering, we use a deep convolutional neural networks (DCNN) classifier and double threshold strategy for classification. Experimental results on public datasets demonstrate that the proposed superpixel based method performs better than MSER in character candidate extraction, and the proposed system achieves competitive performance compared to state-of-the-art methods.

The Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) has been demonstrated successful in handwritten text recognition of Western and Arabic scripts. It is totally segmentation free and can be trained directly from text line images. However, the application of LSTM-RNNs (including Multi-Dimensional LSTM-RNN (MDLSTM-RNN)) to Chinese text recognition has shown limited success, even when training them with large datasets and using pre-training on datasets of other languages. In this paper, we propose a handwritten Chinese text recognition method by using Separable MDLSTMRNN (SMDLSTM-RNN) modules, which extract contextual information in various directions, and consume much less computation efforts and resources compared with the traditional MDLSTMRNN. Experimental results on the ICDAR-2013 competition dataset show that the proposed method performs significantly better than the previous LSTM-based methods, and can compete with the state-of-the-art systems.

Page segmentation is a fundamental and challenging task in document image analysis due to the layout diversity. In this work, we propose a pixel-wise segmentation method for historical handwritten documents using fully convolutional network (FCN). The document image is segmented into different regions by classifying pixels into different categories: background, main text body, comments, and decorations. By supervised learning on document images with pixel-wise labels, the FCN can extract discriminative features and perform pixel-wise segmentation accurately. After pixel-wise classification, post-processing steps are taken to reduce noises, correct wrong segmentations and find out overlapping regions. Experimental results on the public dataset DIVA-HisDB containing challenging medieval manuscripts demonstrate the effectiveness and superiority of the proposed method, which yields pixel-level accuracy of above 99%.

Scene text detection in videos has many application needs but has drawn less attention than that in images. Existing methods for video text detection perform unsatisfactorily because of the insufficient utilization of spatial and temporal information. In this paper, we propose a novel video text detection method with network flow based tracking. The system first applies a newly proposed Fully Convolutional Neural Network (FCN) based scene text detection method to detect texts in individual frames and then track proposals in adjacent frames with a motion-based method. Next, the text association problem is formulated into a cost-flow network and text trajectories are derived from the network with a min-cost flow algorithm. At last, the trajectories are post-processed to improve the precision accuracy. The method can detect multi-oriented scene text in videos and incorporate spatial and temporal information efficiently. Experimental results show that the method improves the detection performance remarkably on benchmark datasets, e.g., by a 15.66% increase of ATA Average Tracking Accuracy) on ICDAR video scene text dataset.

In handwriting recognition, the test data usually come from multiple writers which are not shown in the training data. Therefore, adapting the base classifier towards the new style of each writer can significantly improve the generalization performance. Traditional writer adaptation methods usually assume that there is only one writer (one style) in the test data, and we call this situation as style-clear adaptation. However, a more common situation is that multiple handwriting styles exist in the test data, which is widely appeared in multi-font documents and handwriting data produced by the cooperation of multiple writers. We call the adaptation in this situation as style-mixture adaptation. To deal with this problem, in this paper, we propose a novel method called K-style mixture adaptation (K-SMA) with the assumption that there are totally K styles in the test data. Specifically, we first partition the test data into K groups (style clustering) according to their style consistency, which is measured by a newly designed style feature that can eliminate class (category) information and keep handwriting style information. After that, in each group, a style transfer mapping (STM) is used for writer adaptation. Since the initial style clustering may be not reliable, we repeat this process iteratively to improve the adaptation performance. The K-SMA model is fully unsupervised which do not require either the class label or the style index. Moreover, the K-SMA model can be effectively combined with the benchmark convolutional neural network (CNN) models. Experiments on the online Chinese handwriting database CASIA-OLHWDB demonstrate that K-SMA is an efficient and effective solution for style-mixture adaptation.

Extracting candidate text connected components (CCs) is critical for CC-based text localization. Based on the observation that text strokes in born-digital images mostly have complete contours and the text pixels have high contrast with the adjacent non-text pixels, we propose a method to extract candidate text CCs by combining text contours and stroke interior regions. After segmenting the image into non-smooth and smooth regions based on local contrast, text contour pixels in non-smooth regions are detached from adjacent non-text pixels by local binarization. Then, obvious non-text contours can be removed according to the spatial relationship of text and non-text contours. While smooth regions include stroke interior regions and non-text smooth regions, some non-text smooth regions can be easily removed because they are not surrounded by candidate text contours. At last, candidate text contours and stroke interior regions are combined to generate candidate text CCs. The CCs undergo CC filtering, text line grouping and line classification to give the text localization result. Experimental results on the born-digital dataset of ICDAR2013 robust reading competition demonstrate the efficiency and superiority of the proposed method.

Natural scene character recognition is challenging due to the cluttered background, which is hard to separate from text. In this paper, we propose a novel method for robust scene character recognition. Specifically, we first use robust principal component analysis (PCA) to denoise character image by recovering the missing low-rank component and filtering out the sparse noise term, and then use a simple Histogram of oriented Gradient (HOG) to perform image feature extraction, and finally, use a sparse representation based classifier for recognition. In experiments on four public datasets, namely the Char74K dataset, ICADAR 2003 robust reading dataset, Street View Text (SVT) dataset and IIIT5K-word dataset, our method was demonstrated to be competitive with the state-of-the-art methods.

Text localization in born-digital images is usually performed using methods designed for scene text images. Based on the observation that text strokes in born-digital images mostly have complete contours and the pixels on the contours have high contrast compared with the adjacent non-text pixels, we propose a method to extract candidate text components using local contrast. First, the image is segmented into smooth and non-smooth regions. After removing non-text smooth regions, the remaining smooth regions are merged with non-smooth regions to form a candidate text image, which is binarized into high-value and low-value connected components (CCs). The CCs undergo CC filtering, line grouping and line classification to give the text localization result. Experimental results on the born-digital dataset of ICDAR2013 robust reading competition demonstrate the efficiency and superiority of the proposed method.

Handwritten Chinese text recognition based on over-segmentation and path search integrating contexts has been demonstrated successful, where language models play an important role. Recently, neural network language models (NNLMs) have shown superiority to back-off N-gram language models (BLMs) in handwriting recognition, but have not been studied in Chinese text recognition system. This paper investigates the effects of NNLMs in handwritten Chinese text recognition and compares the performance with BLMs. We trained character-level language models in 3-, 4- and 5- gram on large scale corpora and applied them in text line recognition system. Experimental results on the CASIA-HWDB database show that NNLM and BLM of the same order perform comparably, and the hybrid model by interpolating NNLM and BLM improves the recognition performance significantly.

Visual gesture recognition enables natural human-machine interaction, and writing characters in gesture can convey rich information of intention. However, the recognition of character strings in gesture is challenging because multiple characters are in a single-stroke trajectory without pen lift information. We propose a lexicon-driven approach for gesture character string recognition. Using a lexicon of words to guide character segmentation and recognition, and meanwhile combining the geometric scores of characters and redundant segments with character classification score, we can achieve fairly high recognition accuracy on one-stroke character strings. For experiments, we collected 1,590 gesture strings in 100 word classes of television channel names, and achieved string-level recognition accuracy over 80% on the test set.

In this article, we consider the problem of simultaneous low-rank recovery and sparse projection. More specifically, a new Robust Principal Component Analysis (RPCA)-based framework called Sparse Projection and Low-Rank Recovery (SPLRR) is proposed for handwriting representation and salient stroke feature extraction. In addition to achieving a low-rank component encoding principal features and identify errors or missing values from a given data matrix as RPCA, SPLRR also learns a similarity-preserving sparse projection for extracting salient stroke features and embedding new inputs for classification. These properties make SPLRR applicable for handwriting recognition and stroke correction and enable online computation. A cosine-similarity-style regularization term is incorporated into the SPLRR formulation for encoding the similarities of local handwriting features. The sparse projection and low-rank recovery are calculated from a convex minimization problem that can be efficiently solved in polynomial time. Besides, the supervised extension of SPLRR is also elaborated. The effectiveness of our SPLRR is examined by extensive handwritten digital repairing, stroke correction, and recognition based on benchmark problems. Compared with other related techniques, SPLRR delivers strong generalization capability and state-of-the-art performance for handwriting representation and recognition.

Scene text extraction, i.e., segmenting text pixels from background, is an important step before the text can be recognized. It is a challenging problem due to the cluttered background and the variation of lighting. In this paper, we propose a seed-based segmentation method that can automatically judge the text polarity, extract seed points of text and background, and segment texts by semi-supervised learning (SSL). First, we estimate the text polarity and the stroke width using gradient local correlation. Then, all the points in the middle of stroke edge pairs satisfying the width and polarity are taken as foreground seeds, and the points in the middle of the edge pairs with opposite polarity are taken as background seeds. The whole image is then segmented into text and background using an SSL algorithm. Owing to the accurate estimate of text polarity and extraction of seed points, the proposed method yields good segmentation performance. Experimental results on the KAIST dataset demonstrate the superiority of the method.

In this paper, we propose an efficient scene text localization method using gradient local correlation, which can characterize the density of pair wise edges and stroke width consistency to get a text confidence map. Gradient local correlation is insensitive to the gradient direction and robust to noise, small character size and shadow. Based on the text confidence map, the regions with high confidence are segmented into connected components (CCs), which are classified to text CCs and non-text CCs using an SVM classifier. Then, the text CCs with similar color and stroke width are grouped into text lines, which are in turn partitioned into words. Experimental results on the ICDAR 2003 text locating competition dataset demonstrate the effectiveness of our method.

Page segmentation is still a challenging problem due to the large variety of document layouts. Methods examining both foreground and background regions are among the most effective to solve this problem. However, their performance is influenced by the implementation of two key steps: the extraction and selection of background regions, and the grouping of background regions into separators. This paper proposes an efficient hybrid method for page segmentation. The method extracts white space rectangles based on connected component analysis, and filters white space rectangles progressively incorporating foreground and background information such that the remaining rectangles are likely to form column separators. Experimental results on the ICDAR2009 page segmentation competition test set demonstrate the effectiveness and superiority of the proposed method.

In overlaid handwriting, multiple characters are written sequentially in the same area. This needs special consideration for segmenting the stroke sequence into characters. We propose a learning-based model for scoring the candidate stroke cuts and segments for online overlaid Chinese handwriting recognition. Based on stroke cut classification using support vector machine (SVM), strokes are grouped into segments, and consecutive segments are concatenated into candidate characters. The likeliness of candidate characters (unary geometry) and the compatibility between adjacent characters (binary geometry) are measured by combining the stroke cut score and the between-segment geometric score, and are integrated with the character classification score and linguistic context for character string recognition. Experiments on a large database of online Chinese handwriting demonstrate the effectiveness of the proposed method.

This paper describes the Chinese handwriting recognition competition held at the 12th International Conference on Document Analysis and Recognition (ICDAR 2013). This third competition in the series again used the CASIA-HWDB/OLHWDB databases as the training set, and all the submitted systems were evaluated on closed datasets to report character-level correct rates. This year, 10 groups submitted 27 systems for five tasks: classification on extracted features, online/offline isolated character recognition, online/offline handwritten text recognition. The best results (correct rates) are 93.89% for classification on extracted features, 94.77% for offline character recognition, 97.39% for online character recognition, 88.76% for offline text recognition, and 95.03% for online text recognition, respectively. In addition to the test results, we also provide short descriptions of the recognition methods and brief discussions on the results.

Perturbation-based recognition is effective to recover the deformation of handwritten characters and improve the recognition performance by generating multiple distortions and selecting a distortion that best restores character deformation. Considering that the characters in a field undergo similar deformation under a consistent style, we proposed style consistent perturbation for handwritten character recognition. By generating multiple distortions for the characters in a field, each distortion style is evaluated at the field level and the uniform distortion style of maximum recognition confidence is selected to give the final result. To overcome the slight deviation from uniform style, we also propose to search the neighborhood distortions from the optimal uniform distortion for higher confidence. The experiments of handwritten Chinese character recognition on multi-writer data show that style consistent perturbation in very short fields outperforms individual character recognition, and neighborhood distortion search yields further improvement.

Modified quadratic discriminant function (MQDF) is a state-of-the-art classifier for handwriting recognition. However, the big gap between accuracies on training and testing sets indicates that MQDF has a good capability to fit training data but the generalization performance is not promising. To solve this problem, we propose a new model called locally smoothed modified quadratic discriminant function (LSMQDF) by smoothing the covariance matrix of each class with its nearest neighbor classes. LSMQDF can be viewed as a regularization to avoid over-fitting. The covariance matrix estimated by local smoothing is more accurate and robust. LSMQDF can be also viewed as an extension of the global smoothing method, namely regularized discriminant analysis (RDA). Experiments on both offline and online Chinese handwriting databases demonstrate that: with local smoothing, the accuracy on training set is decreased (over-fitting avoided), and the accuracy on testing set is improved significantly and consistently (generalization improved).

Semi-Markov conditional random fields (semi-CRFs) are usually trained with maximum a posteriori (MAP) criterion which adopts the 0/1 cost for measuring the loss of misclassification. In this paper, based on our previous work on handwritten Chinese/Japanese text recognition (HCTR) using semi-CRFs, we propose an alternative parameter learning method by minimizing the risk, in which the misclassification costs are not equal, but different depending on the hypothesis and the ground-truth. The proposed method is lattice-based, i.e., the hypothesis space is the entire lattice on which the semi-CRF is defined. Experimental results on two online handwriting databases: CASIA-OLHWDB and TUAT Kondate demonstrate that minimum-risk training can yield superior string recognition rates compared to MAP training.

The discriminative training of classifiers for handwritten Chinese character recognition (HCCR) is highly demanding in computation due to the large number of categories. The inability of discriminative training with large sample set on personal computers has hindered the accuracy promotion for HCCR. To overcome this problem, we have implemented the training algorithm of discriminative learning quadratic discriminant function (DLQDF) on our graphics processing units (GPU) server, and have achieved 15 times speedup compared to single-core computation. By enlarging training sample set via distortion on a standard dataset of 3,755 classes, we could train the DLQDF on more than 50 million samples within 150min and get the test accuracy improved by 1.36%.

The well-known feature transformation model of Fisher linear discriminant analysis (FDA) can be decomposed into an equivalent two-step approach: whitening followed by principal component analysis (PCA) in the whitened space. By proving that whitening is the optimal linear transformation to the Euclidean space in the sense of minimum log-determinant divergence, we propose a transformation model called class conditional decor relation (CCD). The objective of CCD is to diagonalize the covariance matrices of different classes simultaneously, which is efficiently optimized using a modified Jacobi method. CCD is effective to find the common principal components among multiple classes. After CCD, the variables become class conditionally uncorrelated, which will benefit the subsequent classification tasks. Combining CCD with the nearest class mean (NCM) classification model can significantly improve the classification accuracy. Experiments on 15 small-scale datasets and one large-scale dataset (with 3755 classes) demonstrate the scalability of CCD for different applications. We also discuss the potential applications of CCD for other problems such as Gaussian mixture models and classifier ensemble learning.

Texts in video provide a rich clue for video indexing and retrieval, yet the detection and recognition of video text remains a challenge. This paper proposes an effective and real-time stroke-based method for text detection in video, which is robust to the change of stroke intensity and width. Particularly, we propose to characterize the text confidence using an edge orientation variance (EOV) and an opposite edge pair (OEP) feature. Based on the text confidence map, candidate text components are extracted and grouped into text lines by thresholding and connected component analysis. Our experimental results demonstrate that the proposed method can detect multilingual texts in video with fairly high accuracy.

This paper presents a robust method for handwritten text line extraction. We use morphological dilation with a dynamic adaptive mask for line extraction. Line separation occurs because of the repulsion and attraction between connected components. The characteristics of the Arabic script are considered to ensure a high performance of the algorithm. Our method is evaluated on the CENPARMI Arabic handwritten documents database which contains multi-skewed and touching lines. With a matching score of 0.95, our method achieved precision and recall rates of 96:3% and 96:7% respectively, which demonstrate the effectiveness of our approach.

This paper investigates the effects of unsupervised language model adaptation (LMA) in handwritten Chinese text recognition. For no prior information of recognition text is available, we use a two-pass recognition strategy. In the first pass, the generic language model (LM) is used to get a preliminary result, which is used to choose the best matched LMs from a set of pre-defined domains, then the matched LMs are used in the second pass recognition. Each LM is compressed to a moderate size via the entropy-based pruning, tree-structure formatting and fewer-byte quantization. We evaluated the LMA for five LM types, including both character-level and word-level ones. Experiments on the CASIA-HWDB database show that language model adaptation improves the performance for each LM type in all domains. The documents of ancient domain gained the biggest improvement of character-level correct rate of 5.87 percent up and accurate rate of 6.05 percent up.

This paper proposes a method for keyword spotting in offline Chinese handwritten documents using a statistical model. On a text query word, the method measures the similarity between the query word and every candidate word in the document by combining a character classifier and four classifiers characterizing the geometric contexts. By over-segmenting text lines into primitive segments, candidate characters and words are generated by concatenating consecutive segments, and the beam search strategy is used to search all the candidate words. The character classifier and the model combining weights are trained by optimizing a one-vs-all discrimination objective so as to maximize the similarity of true words and minimize the similarity of imposters. In experiments on a test dataset containing 1,015 pages of 180 writers, the proposed methods yields promising performance. For retrieving four-characer words, the recall, precision and F-measure are 92.47%, 83.76% and 87.90%, respectively.

This paper introduces a pair of online and offline Chinese handwriting databases, containing samples of isolated characters and handwritten texts. The samples were produced by 1,020 writers using Anoto pen on papers for obtaining both online trajectory data and offline images. Both the online samples and offline samples are divided into six datasets, three for isolated characters (DB1.0-C1.2) and three for handwritten texts (DB2.0-C2.2). The (either online or offline) datasets of isolated characters contain about 3.9 million samples of 7,356 classes (7,185 Chinese characters and 171 symbols), and the datasets of handwritten texts contain about 5,090 pages and 1.35 million character samples. Each dataset is segmented and annotated at character level, and is partitioned into standard training and test subsets. The online and offline databases can be used for the research of various handwritten document analysis tasks.

In the Chinese handwriting recognition competition organized with the ICDAR 2011, four tasks were evaluated: offline and online isolated character recognition, offline and online handwritten text recognition. To enable the training of recognition systems, we announced the large databases CASIA-HWDB/OLHWDB. The submitted systems were evaluated on un-open datasets to report character-level correct rates. In total, we received 25 systems submitted by eight groups. On the test datasets, the best results (correct rates) are 92.18% for offline character recognition, 95.77% for online character recognition, 77.26% for offline text recognition, and 94.33% for online text recognition, respectively. In addition to the evaluation results, we provide short descriptions of the recognition methods and have brief discussions.

Modified quadratic discriminant function (MQDF) is the state-of-the-art classifier in handwritten character recognition. Discriminative learning of MQDF can further improve its performance. Recent advances justify the efficacy of minimum classification error criteria in learning MQDF (MCE-MQDF). We provide an alternative choice to MCE-MQDF based on the Perceptron learning (PL-MQDF). For better generalization performance, we propose a new dynamic margin regularization. To relieve the heavy burden in training process, active set technique is employed, which can save most of the computation with negligible loss in accuracy. In experiments on handwritten digit datasets and a large-scale Chinese handwritten character database, the proposed PL-MQDF was demonstrated superior in both error reduction and training speedup.

Real-time recognition of handwritten sentences enables fast text input but the dynamic nature of writing makes reliable text line segmentation difficult. This paper proposes a method for real-time dynamic text line segmentation of online Chinese handwriting. The core of the method is a statistical classifier for modeling the geometric relationship between an ongoing stroke and the previous text lines, to assign the stroke into a previous line or form a new line. The method can deal with delayed strokes and therefore enables robust real-time recognition. We evaluated the segmentation performance on a dataset of online Chinese handwriting by simulating the real-time writing and recognition process. The experimental results demonstrate the effectiveness and robustness of the proposed method.

This paper investigates the effects of confidence transformation (CT) of the character classifier outputs in handwritten Chinese text recognition. The classifier outputs are transformed to confidence values in three confidence types, namely, sigmoid, soft max and Dempster-Shafer theory of evidence (D-S evidence). The confidence parameters are optimized by minimizing the cross-entropy (CE) loss function (both binary and multi-class) on a validation dataset, where we add non-character samples to enhance the outlier rejection capability in text recognition. Experimental results on the CASIA-HWDB database show that confidence transformation improves the handwritten text recognition performance significantly and adding non-characters for confidence parameter estimation is beneficial. Among the confidence types, the D-S evidence performs best.

In offline handwritten text recognition, the separation of touching characters remains a challenge due to the variability of touching structures. This paper proposes a new touching character separation method for Chinese handwriting based on skeleton analysis and contour analysis incorporating the visibility of separating points. Separating points are detected from strokes that are common in both upper and lower skeleton tracing, and the profile visibility of strokes and separating points is analyzed to adjust and verify separating points. Our experiments on two large handwriting databases demonstrate the effectiveness of the proposed method.

This paper presents a conditional random field (CRF) model for aligning online handwritten Chinese/Japanese text lines (character strings) with the corresponding transcripts. The CRF model is defined on a lattice which contains all possible segmentation hypotheses. The feature functions characterize the shape and context dependences of characters, including the scores of character recognition and the geometric compatibilities between characters. The combining parameters are optimized by energy minimization. Experimental results on two online databases: CASIA-OLHWDB and TUAT Kondate demonstrate the effectiveness of the proposed method.

In this paper, we propose an efficient and robust algorithm for graph-based transductive classification. After approximating a graph with a spanning tree, we develop a linear-time algorithm to label the tree such that the cut size of the tree is minimized. This significantly improves typical graph-based methods, which either have a cubic time complexity (for a dense graph) or O(kn2) (for a sparse graph with k denoting the node degree). Furthermore, our method shows great robustness to the graph construction both theoretically and empirically; this overcomes another big problem of traditional graph-based methods. In addition to its good scalability and robustness, the proposed algorithm demonstrates high accuracy. In particular, on a graph with 400,000 nodes (in which 10,000 nodes are labeled) and 10,455,545 edges, our algorithm achieves the highest accuracy of 99.6% but takes less than 10 seconds to label all the unlabeled data.

In this paper, we present a semi-supervised method to learn a low rank Mahalanobis distance function. Based on an approximation to the projection distance from a manifold, we propose a novel parametric manifold regularizer. In contrast to previous approaches that usually exploit side information only, our proposed method can further take advantages of the intrinsic manifold information from data. In addition, we focus on learning a metric of low rank directly, this is different from traditional approaches that often enforce the l1norm on the metric. The resulting configuration is convex with respect to the manifold structure and the distance function, respectively. We solve it with an alternating optimization algorithm, which proves effective to find a satisfactory solution. For efficient implementation, we even present a fast algorithm, in which the manifold structure and the distance function are learned independently without alternating minimization. Experimental results over 12 standard UCI data sets demonstrate the advantages of our method.

This paper proposes a novel hybrid method to robustly and accurately localize texts in natural scene images. A text region detector is designed to generate a text confidence map, based on which text components can be segmented by local binarization approach. A conditional random field (CRF) model, considering the unary component property as well as binary neighboring component relationship, is then presented to label components as "text" or "non-text". Last, text components are grouped into text lines with an energy minimization approach. Experimental results show that the proposed method gives promising performance comparing with the existing methods on ICDAR 2003 competition dataset.

This paper describes a publicly available database, CASIA-OLHWDB1, for research on online handwritten Chinese character recognition. This database is the first of our series of online/offline handwritten characters and texts, collected using Anoto pen on paper. It contains unconstrained handwritten characters of 4,037 categories (3,866 Chinese characters and 171 symbols) produced by 420 persons, and 1,694,741 samples in total. It can be used for design and evaluation of character recognition algorithms and classifier design for handwritten text recognition systems. We have partitioned the samples into three grades and into training and test sets. Preliminary experiments on the database using a state-of-the-art recognizer justify the challenge of recognition.

This paper describes a system for handwritten Chinese text recognition integrating language model. On a text line image, the system generates character segmentation and word segmentation candidates, and the candidate paths are evaluated by character recognition scores and language model. The optimal path, giving segmentation and recognition result, is found using a pruned dynamic programming search method. We evaluate various language models, including the character-based n-gram, word-based n-gram, and hybrid n-gram models. Experimental results on the HIT-HW database show that the language models improve the recognition performance remarkably.

Text line segmentation in unconstrained handwritten documents remains a challenge because handwritten text lines are multi-skewed and not obviously separated. This paper presents a new approach based on the variational Bayes (VB) framework for text line segmentation. Viewing the document image as a mixture density model, with each text line approximated by a Gaussian component, the VB method can automatically determine the number of components. We extend the VB method such that it can both eliminate and split components and control the orientation of text line lines. Experiments on Chinese handwritten documents demonstrated the effectiveness of the approach.

Annotating the regions, text lines and characters of document images is an important, but tedious and expensive task. A ground-truthing tool may largely alleviate the human burden in this process. This paper describes an automated recognition-based tool GTLC for finding the best alignment between the text transcript and the connected components of unconstrained handwritten document image. The alignment process is formulated as an optimization problem involving candidate character segmentation and recognition. We have validated the effectiveness of this tool and have used it for annotating a large number of handwritten Chinese documents.

In this paper, we present a robust system to accurately detect and localize texts in natural scene images. For text detection, a region-based method utilizing multiple features and cascade AdaBoost classifier is adopted. For text localization, a window grouping method integrating text line competition analysis is used to generate text lines. Then within each text line, local binarization is used to extract candidate connected components (CCs) and non-text CCs are filtered out by Markov Random Fields (MRF) model, through which text line can be localized accurately. Experiments on the public benchmark ICDAR 2003 Robust Reading and Text Locating Dataset show that our system is comparable to the best existing methods both in accuracy and speed.

We present an effective approach for grouping text lines in online handwritten Japanese documents by combining temporal and spatial information. Initially, strokes are grouped into text line strings according to off-stroke distances. Each text line string is segmented into text lines by dynamic programming (DP) optimizing a cost function trained by the minimum classification error (MCE) method. Over-segmented text lines are then merged with a support vector machine (SVM) classifier for making merge/non-merge decisions, and last, a spatial merge module corrects the segmentation errors caused by delayed strokes. In experiments on the TUAT Kondate database, the proposed approach achieves the Entity Detection Metric (EDM) rate of 0.8816, the Edit-Distance Rate (EDR) of 0.1234, which demonstrates the superiority of our approach.