Deep learning-based sequence modeling methods have improved the performance in Chinese handwriting recognition tasks. However, the implicit representations learned in current deep neural network models usually lack explainability and generalization ability for practical handwriting samples with diverse styles. This paper proposes a disentangled representation learning method for Chinese handwriting recognition by using a latent Gaussian process, which models the features in latent space as Gaussian distributions conditioned on the content and style of handwriting samples. A deep learning model with one encoder and multiple decoders is designed. The style embeddings in a style decoder are used to convert the hidden representations output by the encoder into style-agnostic representations for content decoding. Objective functions to decouple content and style representations are introduced in the training stage. Experiments are conducted on the public CASIA-HWDB Chinese handwriting dataset and a self-built historical Chinese local genealogies dataset, which demonstrate that the proposed method achieves high accuracy in recognizing handwritten Chinese text images with diverse styles.

Deep learning models for handwriting recognition have been developed in recent years. To improve the model’s generalization ability for sequence modeling task, this paper proposes to use domain adaptation with statistical distribution alignment and entropy regularization. For statistical distribution alignment, a domain adaptation loss function is proposed by using both the first and second order statistical information of deep feature representations, which is equivalent to maximizing the mutual information in feature spaces of the source domain and target domain. For entropy regularization, the entropy of the predicted text symbols of unlabeled samples in the target domain is also utilized as an additional loss function, which maximizes the mutual information between the feature space and pattern space in the target domain. Experimental results on the IAM handwriting dataset have demonstrated the effectiveness of the proposed domain adaptation method for sequence modeling task.

Scene text recognition is a challenging task due to di-verse variations of text instances in natural scene images. Conventional methods based on CNN-RNN-CTC or encoder-decoder with attention mechanism may not fully investigate stable and efficient feature representations for multi-oriented scene texts. In this paper, we propose a primitive representation learning method that aims to exploit intrinsic representations of scene text images. We model elements in feature maps as the nodes of an undirected graph. A pooling aggregator and a weighted aggregator are proposed to learn primitive representations, which are transformed into high-level visual text representations by graph convolutional networks. A Primitive REpresentation learning Network (PREN) is constructed to use the visual text representations for parallel decoding. Furthermore, by integrating visual text representations into an encoder-decoder model with the 2D attention mechanism, we pro-pose a framework called PREN2D to alleviate the misalignment problem in attention-based methods. Experimental results on both English and Chinese scene text recognition tasks demonstrate that PREN keeps a balance between ac-curacy and efficiency, while PREN2D achieves state-of-the-art performance.

Long short-term memory (LSTM) networks have been widely used in automatic speech recognition (ASR). This paper proposes a novel dynamic temporal residual learning mechanism for LSTM networks to better explore temporal dependencies in sequential data. The temporal residual learning mechanism is implemented by applying shortcut connections with dynamic weights to temporally adjacent LSTM outputs. Two types of dynamic weight generation methods are proposed: using a secondary network and using a random weight generator. Experimental results on Wall Street Journal (WSJ) speech recognition dataset reveal that our proposed methods have surpassed the baseline LSTM network.

Scene text recognition is a challenging problem due to the wide variances in contents, styles, orientations, and image quality of text instances in natural scene images. To learn the intrinsic representation of scene texts, a novel multi-element attention (MEA) mechanism is proposed to exploit geometric structures from local to global levels in feature maps extracted from a scene text image. The MEA mechanism is a generalized form of self-attention technique. The elements in feature maps are taken as the nodes of an undirected graph, and three kinds of adjacency matrices are designed to aggregate information at local, neighborhood and global levels before calculating the attention weights. A multi-element attention network (MEAN) is implemented, which includes a CNN for feature extraction, an encoder with MEA mechanism and a decoder for predicting text codes. Orientational positional encoding is added to feature maps output by the CNN, and a feature vector sequence transformed from the feature maps is used as the input of the encoder. Experimental results show that MEAN has achieved state-of-the-art or competitive performance on seven public English scene text datasets (IIITSk, SVT, IC03, IC13, IC15, SVTP, and CUTE). Further experiments have been conducted on a selected subset of the RCTW Chinese scene text dataset, demonstrating that MEAN can handle horizontal, vertical, and irregular scene text samples.

Historical Chinese character recognition faces problems including low image quality and lack of labeled training samples. We propose a generative adversarial network (GAN) based transfer learning method to ease these problems. The proposed TH-GAN architecture includes a discriminator and a generator. The network structure of the discriminator is based on a convolutional neural network (CNN). Inspired by Wasserstein GAN, the loss function of the discriminator aims to measure the probabilistic distribution distance of the generated images and the target images. The network structure of the generator is a CNN based encoder-decoder. The loss function of the generator aims to minimize the distribution distance between the real samples and the generated samples. In order to preserve the complex glyph structure of a historical Chinese character, a weighted mean squared error (MSE) criterion by incorporating both the edge and the skeleton information in the ground truth image is proposed as the weighted pixel loss in the generator. These loss functions are used for joint training of the discriminator and the generator. Experiments are conducted on two tasks to evaluate the performance of the proposed TH-GAN. The first task is carried out on style transfer mapping for multi-font printed traditional Chinese character samples. The second task is carried out on transfer learning for historical Chinese character samples by adding samples generated by TH-GAN. Experimental results show that the proposed TH-GAN is effective.

Handwritten Chinese character recognition (HCCR) is a representative large character set pattern classification task. Recently, convolutional neural networks have provided promising solutions for this challenging task. This paper adopts the modified Inception-ResNet network for handwritten Chinese character recognition, and proposes a discriminant weighting method for cross-entropy loss calculation which focuses on recognition errors in the training stage. Sparse training technique is also incorporated. Under the specific condition of utilizing the testing mini-batch mean and variance for batch normalization, the proposed method achieves improved performance on the ICDAR-2013 offline handwritten Chinese character competition dataset.

Offline handwriting recognition is a well-known challenging task in the optical character recognition (OCR) field due to the difficulty caused by various unconstraint handwriting styles. In order to learn invariant feature representations for handwriting, we propose a novel method to incorporate pixel-level rectification into a CNN and RNN based model. We also propose an adjacent output mixup method for RNN layer's training to improve the generalization ability of the model, i.e., the previous output of an RNN layer is added to the current output with random weights. We additionally adopt a series of techniques including pre-training, data augmentation and language model, and further analyze their contributions to the improvement of the model performance. The proposed method performs well on three public benchmarks, including the IAM, Rimes and IFN/ENIT datasets.

Document Image Quality Assessment (DIQA) is an essential step preceding Optical Character Recognition (OCR). In this paper we propose an attention based Recurrent Neural Network (RNN) model for camera based DIQA. Convolutional Neural Network (CNN) and RNN are integrated into our model to capture spatial features for several glimpse regions step by step within an image patch. Reinforcement learning is adopted to train a locator to generate the optimal location of a glimpse region for the next time step so that attention can be payed to the salient part. Given an input document image, patches are generated with a sliding window, and the pure background ones are sifted out. Quality scores are obtained for all the sifted patches by applying the proposed attention based RNN method, and the patch scores are averaged over each input image as the result of DIQA. We conduct experiments on two public datasets and make comparisons with several other reported methods. Experimental results show that our model achieves the state of the art performance.

Although transfer learning has aroused researchers' great interest, how to utilize the unlabeled data is still an open and important problem in this area. We propose a novel semi-supervised transfer learning (STL) method by incorporating Multi-Kernel Maximum Mean Discrepancy (MK-MMD) loss into the traditional fine-tuned Convolutional Neural Network (CNN) transfer learning framework for Chinese character recognition. The proposed method includes three steps. First, a CNN model is trained by massive labeled samples in the source domain. Then the CNN model is fine-tuned by a few labeled samples in the target domain. Finally, the CNN model is trained with both a large number of unlabeled samples and the limited labeled samples in the target domain to minimize the MK-MMD loss. Experiments investigate detailed configurations and parameters of the proposed STL method with several frequently used CNN structures including AlexNet, GoogLeNet, and ResNet. Experimental results on practical Chinese character transfer learning tasks, such as Dunhuang historical Chinese character recognition, indicate that the proposed method can significantly improve recognition accuracy in the target domain.

Deep Recurrent Neural Networks (RNN) have been suffering from the overfitting problem due to the model redundancy of the network structures. We propose a novel temporal and spatial residual learning method for RNN, followed with sparse training by weight pruning to gain sparsity in network parameters. For a Long Short-Term Memory (LSTM) network, we explore the combination schemes and parameter settings for temporal and spatial residual learning with sparse training. Experiments are carried out on the IFN/ENIT database. For the character error rate on the testing set e while training with sets a, b, c, d, the previously reported best result is 13.42%, and the proposed configuration of temporal residual learning followed with sparse training achieves the state-of-the-art result 12.06%.

This paper investigates local discriminant training and global optimization methods for Convolutional Neural Network (CNN) to improve its discriminant ability and recognition accuracy. For local discriminant training, we propose to combine triplet loss and softmax with cross-entropy loss as the loss function. The triplet loss is incorporated into an additional fully-connected layer before the final fully-connected layer of a CNN model. For global optimization, we use Conditional Random Field (CRF) to further utilize the pairwise distance of the CNN feature vectors trained with triplet loss. Experiments with different CNN models on handwritten Chinese character samples show that the combined local discriminant training and global optimization scheme achieves better character recognition accuracy and confidence analysis performance.

Uyghur language is written in a modified Arabic script. Due to its cursive nature and the lack of enough labeled training samples, Uyghur document recognition is still a challenging problem. In this paper, we propose a new Recurrent Neural Network (RNN) based Uyghur text line recognition method combining Gated Recurrent Unit (GRU) and Restricted Boltzmann Machine (RBM) with pretraining mechanism. We also present a novel curriculum learning technique guided by sample distribution information. Experimental results on practical Uyghur printed document image dataset show that the proposed network architecture and training strategy not only achieve better recognition accuracy compared with traditional methods, but can accelerate the training speed as well.

Historical Chinese character recognition has been suffering from the problem of lacking sufficient labeled training samples. A transfer learning method based on Convolutional Neural Network (CNN) for historical Chinese character recognition is proposed in this paper. A CNN model L is trained by printed Chinese character samples in the source domain. The network structure and weights of model L are used to initialize another CNN model T, which is regarded as the feature extractor and classifier in the target domain. The model T is then fine-tuned by a few labeled historical or handwritten Chinese character samples, and used for final evaluation in the target domain. Several experiments regarding essential factors of the CNNbased transfer learning method are conducted, showing that the proposed method is effective.

With the rapid increase of on-line video resources, there is an urgent demand for text detection and recognition technologies to build content-based video indexing and retrieval systems. Chinese news video texts contain highly condensed and rich information, but the low resolution of videos on the Internet and the complexity of Chinese character structures bring challenges for text detection. In this paper, we present a multi-stage scheme for Chinese news video text detection. We propose an improved Stroke Width Transform (SWT) method by incorporating text color consistency constraint for candidate text blocks generation. Then we use “divide and conquer” strategy to distinguish candidate text blocks into three sub-spaces according to their geometric shapes and size. For each sub-space, a neural network is designed to filter the candidates into text or non-text blocks. Finally, the text blocks are merged into text lines based on the stroke width, color and other heuristic information. Experimental results on self-collected Chinese news video dataset and ICDAR 2013 dataset show that the proposed method is effective to detect both news video captions and scene texts.

Hidden Markov Model (HMM) is an effective method to describe sequential signals in many applications. As to model estimation issue, common training algorithm only focuses on the optimization of model parameters. However, model structure influences system performance as well. Although some structure optimization methods are proposed, they are usually implemented as an independent module before parameter optimization. In this paper, the clustering feature of states in HMM is discussed through comparing the mechanism of Quadratic Discriminant Function (QDF) classifier and HMM. Then, through the clustering effect of Viterbi training and Baum–Welch training, a novel clustering-based model pre-training approach is proposed. It can optimize model parameters and model structure by turns, until the representative states of all models are explored. Finally, the proposed approach is evaluated on two typical OCR applications, printed and handwritten Arabic text line recognition. And it is compared with some other optimization methods. The improvement of character recognition performance proves the proposed approach can make more precise state allocation. And the representative states are benefit to HMM decoding.

Recognition of handwritten mathematical expressions (HMEs) has become a cutting edge research topic recently, as there are increasingly needs for pen-inputting applications. In this paper, we presented a novel framework to analyse HME layout and semantic information. This framework includes three steps, namely symbol segmentation, symbol recognition and semantic relationship analysis. For symbol segmentation, a decomposition on strokes is operated, then dynamic programming is adopted to find the paths corresponding to the best segmentation manner and reduce the stroke searching complexity. For symbol recognition, spatial geometry and directional element features are classified by a Gaussian Mixture Model learnt through Expectation-Maximization algorithm. At last, in the semantic relationship analysis module, a ternary tree is utilized to to store the ranked symbols through calculating the operator priorities. The motivation for our work comes from the apparent difference in writing styles across western and Chinese populations. Our results are reasonable and show promise on the private dataset.

Historical Chinese document recognition technology is important for digital library. However, historical Chinese character segmentation remains a difficult problem due to the complex structure of Chinese characters and various writing styles. This paper presents a novel method for historical Chinese character segmentation based on graph model. After a preliminary over-segmentation stage, the system applies a merging process. The candidate segmentation positions are denoted by the nodes of a graph, and the merging process is regarded as selecting an optimal path of the graph. The weight of edge in the graph is calculated by the cost function which considers geometric features and recognition confidence. Experimental results show that the proposed method is effective with a detection rate of 94.6% and an accuracy rate of 96.1% on a test set of practical historical Chinese document samples.

Historical Chinese character recognition has been a challenging topic in pattern recognition field because of large character set, various writing styles and lack of training samples. In this paper, we adopted Style Transfer Mapping (STM) method to historical Chinese character recognition. Optimal selection of parameters was discussed. Two sets of experiments were conducted. The first set of experiment was designed to test the performance of STM on different font styles by using available printed traditional Chinese characters. The second set of experiment was carried out on samples extracted from practical historical Chinese documents. Experimental results showed that supervised STM may improve the generalization ability of the classifier.

This paper proposes a method to recognize stroke orders of on-line handwritten Chinese characters based on analyzing both spatial and temporal information. A novel control-point-based analysis method is presented for spatial information analysis to match strokes with various shapes and styles. Its computation complexity is much lower than image correlation method and is suitable for applications on mobile devices. For temporal information analysis, Hidden Markov Model is adopted and a proposed rectification method is integrated to find the optimal pair-wise matching result of stroke sequences. Experimental results proved the effectiveness of the proposed method. The verification rate is 99.6% on the test set.

Cross-language document recognition and understanding have urgent realistic needs and extensive application prospects. In this paper, we propose a novel recognition-based Uighur and Tibetan document understanding method, termed "cross-language sensitive words distribution map" (CSWDM). In our unified recognition-understanding framework, digital Uighur/Tibetan document images are first recognized using OCR technology, and then CSWDM labels the Chinese information of sensitive words on the recognized transcriptions or directly on the original digital images, thus the space location and occurrence frequency of these sensitive words can be intuitively represented. With such information, readers can roughly understand the theme and meaning of the cross-language documents.

HMM-based analytical methods have been widely used for Arabic handwriting recognition. A key factor influencing the performance of HMM-based systems is the features extracted from a sliding window. In this paper, we propose a novel baseline-independent feature set extracted from a wider sliding window to directly capture the contextual information. This feature set is a combination of center of mass based log-space distribution features and inverse percentile features. Center of mass based log-space distribution features use a normalized histogram to describe the distribution of foreground pixels in different direction and distances with respect to the center of mass. Experiments on the IFN/ENIT database demonstrate the effectiveness of the proposed feature set. Further, this feature set can be combined with some popular baseline-independent features to form a large feature set, which achieves comparable results with several state-of-the-art systems using a simple HMM-based architecture.

Writer recognition is a very important branch of biometrics. In our previous research, a Grid Micro-structure Feature (GMSF) based text-independent and script-independent method was adopted and high performance was obtained. However, this method is sensitive to pen-width variation in practical situation. To solve this problem, an inner and inter class variances weighted high-dimensional feature matching method is proposed. The inner and inter class variances are estimated on handwriting samples with different pen-width written by different writers. Experimental results show that our method is effective.

Automatic performance evaluation for a document page segmentation module is necessary, as OCR products are used to manipulate large scale of documents with complex layout, especially for newspapers. The paper presents a region-based method to evaluate the performance of a page segmentation module by analyzing geometric region relationships between the segmentation results and the preset ground-truth. The ground-truth is not only the correct answer to page segmentation, but also the comparison benchmark of the automatic evaluation, so it has more restricted geometric constraints. The region-matching algorithm is realized by searching the equal region in the segmentation results for each region in the ground-truth. The performance parameters are calculated based on the matching results. An experiment is given to test two page segmentation modules in a popular Chinese OCR product-THOCR2000, and the results show this method is effective.