Current multimodal large language models (MLLMs) face significant challenges in visual document understanding (VDU) tasks due to the high resolution, dense text, and complex layouts typical of document images. These characteristics demand a high level of detail perception ability from MLLMs. While increasing input resolution improves detail perception capability, it also leads to longer sequences of visual tokens, increasing computational costs and straining the models' ability to handle long contexts. To address these challenges, we introduce DocKylin, a document-centric MLLM that performs visual content slimming at both the pixel and token levels, thereby reducing token sequence length in VDU scenarios. We introduce an Adaptive Pixel Slimming (APS) preprocessing module to perform pixel-level slimming, increasing the proportion of informative pixels. Moreover, we propose a novel Dynamic Token Slimming (DTS) module to conduct token-level slimming, filtering essential tokens and removing others to adaptively create a more compact visual sequence. Experiments demonstrate DocKylin's promising performance across various VDU benchmarks and the effectiveness of each component.

Document layout analysis is a crucial prerequisite for document understanding, including document retrieval and conversion. Most public datasets currently contain only PDF documents and lack realistic documents. Models trained on these datasets may not generalize well to real-world scenarios. Therefore, this paper introduces a large and diverse document layout analysis dataset called M6Doc. The M6 designation represents six properties: (1) Multi-Format (including scanned, photographed, and PDF documents); (2) Multi-Type (such as scientific articles, textbooks, books, test papers, magazines, newspapers, and notes); (3) Multi-Layout (rectangular, Manhattan, non-Manhattan, and multi-column Manhattan); (4) Multi-Language (Chinese and English); (5) Multi-Annotation Category (74 types of annotation labels with 237,116 annotation instances in 9,080 manually annotated pages); and (6) Modern documents. Additionally, we propose a transformer-based document layout analysis method called TransDLANet, which leverages an adaptive element matching mechanism that enables query embedding to better match ground truth to improve recall, and constructs a segmentation branch for more precise document image instance segmentation. We conduct a comprehensive evaluation of M6Doc with various layout analysis methods and demonstrate its effectiveness. TransDLANet achieves state-of-the-art performance on M6 Doc with 64.5% mAP. The M6Doc dataset will be available at https://github.com/HcIILAB/M6Doc.

Table structure recognition aims to extract the logical and physical structure of unstructured table images into a machine-readable format. The latest end-to-end image-to-text approaches simultaneously predict the two structures by two decoders, where the prediction of the physical structure (the bounding boxes of the cells) is based on the representation of the logical structure. However, the previous methods struggle with imprecise bounding boxes as the logical representation lacks local visual information. To address this issue, we propose an end-to-end sequential modeling framework for table structure recognition called VAST. It contains a novel coordinate sequence decoder triggered by the representation of the non-empty cell from the logical structure decoder. In the coordinate sequence decoder, we model the bounding box coordinates as a language sequence, where the left, top, right and bottom coordinates are decoded sequentially to leverage the inter-coordinate dependency. Furthermore, we propose an auxiliary visual-alignment loss to enforce the logical representation of the non-empty cells to contain more local visual details, which helps produce better cell bounding boxes. Extensive experiments demonstrate that our proposed method can achieve state-of-the-art results in both logical and physical structure recognition. The ablation study also validates that the proposed coordinate sequence decoder and the visual-alignment loss are the keys to the success of our method.

Most current detection methods have adopted anchor boxes as regression references. However, the detection performance is sensitive to the setting of the anchor boxes. A proper setting of anchor boxes may vary significantly across different datasets, which severely limits the universality of the detectors. To improve the adaptivity of the detectors, in this paper, we present a novel dimension-decomposition region proposal network (DeRPN) that can perfectly displace the traditional Region Proposal Network (RPN). DeRPN utilizes an anchor string mechanism to independently match object widths and heights, which is conducive to treating variant object shapes. In addition, a novel scale-sensitive loss is designed to address the imbalanced loss computations of different scaled objects, which can avoid the small objects being overwhelmed by larger ones. Comprehensive experiments conducted on both general object detection datasets (Pascal VOC 2007, 2012 and MS COCO) and scene text detection datasets (ICDAR 2013 and COCO-Text) all prove that our DeRPN can significantly outperform RPN. It is worth mentioning that the proposed DeRPN can be employed directly on different models, tasks, and datasets without any modifications of hyperparameters or specialized optimization, which further demonstrates its adaptivity. The code has been released at https://github.com/HCIILAB/DeRPN.

Evaluation protocols play key role in the developmental progress of text detection methods. There are strict requirements to ensure that the evaluation methods are fair, objective and reasonable. However, existing metrics exhibit some obvious drawbacks: 1) They are not goal-oriented; 2) they cannot recognize the tightness of detection methods; 3) existing one-to-many and many-to-one solutions involve inherent loopholes and deficiencies. Therefore, this paper proposes a novel evaluation protocol called Tightness-aware Intersect-over-Union (TIoU) metric that could quantify completeness of ground truth, compactness of detection, and tightness of matching degree. Specifically, instead of merely using the IoU value, two common detection behaviors are properly considered; meanwhile, directly using the score of TIoU to recognize the tightness. In addition, we further propose a straightforward method to address the annotation granularity issue, which can fairly evaluate word and text-line detections simultaneously. By adopting the detection results from published methods and general object detection frameworks, comprehensive experiments on ICDAR 2013 and ICDAR 2015 datasets are conducted to compare recent metrics and the proposed TIoU metric. The comparison demonstrated some promising new prospects, e.g., determining the methods and frameworks for which the detection is tighter and more beneficial to recognize. Our method is extremely simple; however, the novelty is none other than the proposed metric can utilize simplest but reasonable improvements to lead to many interesting and insightful prospects and solving most the issues of the previous metrics. The code is publicly available at https://github.com/Yuliang-Liu/TIoU-metric.

In this paper, we propose a novel method, aggregation cross-entropy (ACE), for sequence recognition from a brand new perspective. The ACE loss function exhibits competitive performance to CTC and the attention mechanism, with much quicker implementation (as it involves only four fundamental formulas), faster inference\back-propagation (approximately O(1) in parallel), less storage requirement (no parameter and negligible runtime memory), and convenient employment (by replacing CTC with ACE). Furthermore, the proposed ACE loss function exhibits two noteworthy properties: (1) it can be directly applied for 2D prediction by flattening the 2D prediction into 1D prediction as the input and (2) it requires only characters and their numbers in the sequence annotation for supervision, which allows it to advance beyond sequence recognition, e.g., counting problem. The code is publicly available at https://github.com/summerlvsong/Aggregation-Cross-Entropy.

To date, offline handwriting paragraph recognition systems either separately crop text line images and recognize them or perform implicit line segmentation by integrating complicated multi-dimensional long short-term memory (MDLSTM) with an attention mechanism. The former abovementioned approachs could lead to sub-optimal performances while the latter is very time-consuming. In this paper, a fast end-to-end system, called adversarial feature enhancing network (AFEN), is proposed for offline handwritten paragraph recognition. The proposed AFEN system comprises five components: a shared feature extractor for robust feature learning, a text detection branch for text box proposal; RoIRotate for oriented feature region extraction, an adversarial feature learning network for joint feature learning of text detection and recognition branch, and a text recognition branch for text transcription. Experiments on two popular handwritten paragraph recognition benchmarks, namely IAM and Rimes are used to verify the efficacy of the proposed AFEN system. The proposed approach yields impressive results compared to previously proposed systems in the literature.

With the passing of history, precious cultural heritage was left behind to tell ancient stories, especially those in the form of written documents. In this paper, a weakly supervised segmentation system with recognition-guided information on attention area, is proposed for high-precision historical document segmentation under strict intersection-over-union (IoU) requirements. We formulate the character segmentation problem from Bayesian decision theory perspective and propose boundary box segmentation (BBS), recognition-guided BBS (Rg-BBS), and recognition-guided attention BBS (Rg-ABBS), progressively, to search for the segmentation path. Furthermore, a novel judgment gate mechanism is proposed to train a high-performance character recognizer in an incremental weakly supervised learning manner. The proposed Rg-ABBS method is shown to substantially reduce time consumption while maintaining sufficiently high precision of the segmentation result by incorporating both character recognition knowledge and line-level annotation. Experiments show that the proposed Rg-ABBS system significantly outperforms traditional segmentation methods as well as deep-learning-based instance segmentation and detection methods under strict IoU requirements.

Offline handwritten Chinese character recognition has attracted much interest due to its various applications. The most cutting-edge methods treat Chinese character as a whole, ignoring the structures and radicals that compose characters. To use the radical-level composition of Chinese characters and achieve few-shot/zero-shot Chinese character recognition, some methods attempt to recognize Chinese characters at the radical level; however, these methods have shown poor performance due to weak radical feature representation and the use of inflexible decoding algorithm. In this paper, a novel radical aggregation network (RAN) is proposed for few-shot/zero-shot offline handwritten Chinese character recognition. The RAN consists of three components, a radical mapping encoder (RME), a radical aggregation module (RAM), and a character analysis decoder (CAD). Experiments show that our method can effectively recognize unseen handwritten characters given few support samples, while maintaining a high performance on seen characters.

Handwritten texts with various styles, such as horizontal, overlapping, vertical, and multi-lines texts, are commonly observed in the community. However, most existing handwriting recognition methods only concentrate on one specific kind of text style. In this paper, we focus on the problem of new unconstrained handwritten text recognition and propose distilling gated recurrent unit (GRU) with a new data augmentation technology to model the complex sequential dynamic of unconstrained handwriting text of various styles. The proposed data augmentation method can synthesize realistic handwritten text datasets including horizontal, vertical, overlap, right-down, screw-rotation, and multi-line situation, which render our framework robust for general purposes. The recommended distilling GRU can not only accelerate the training speed through the distilling stage but also maintain the original recognition accuracy. Experiments on our synthesized handwritten test sets show that the proposed multi-layer GRU performs well on the unconstrained handwriting text recognition problem. On the ICDAR2013 handwritten text recognition benchmark dataset, the proposed framework demonstrates comparable performance with state-of-the-art techniques.

With the development of deep learning tools, the online handwritten Chinese character recognition (HCCR) performance has been greatly improved by using deep neural networks (DNNs) especially for long short-term memory (LSTM). However, DNNs suffer from large consumption of computation and storage resources, which may cause problems for service providers, such as server pressure, longer service latency and higher energy consumption. To solve these problems, we propose a framework that combines singular value decomposition (SVD) and adaptive drop weight (ADW) to accelerate and compress LSTM based models. We first build an LSTM based model that achieves an accuracy of 97.83% on the ICDAR2013 online HCCR competition dataset. After restructuring the model with SVD and ADW, it can reduce the FLOPs (floating point operations per second) of the forward process by approximately 10 times and compress the model with 1/30 of the original size with only a 0.5% decrease in accuracy. Finally, integrated with our efficient forward implementation, the recognition of an online character requires only 2.7 ms in average on a CPU with a single thread, while requiring only 0.45 MB for model storage.

Online handwritten Chinese text recognition (OHCTR) is a challenging problem as it involves a large-scale character set, ambiguous segmentation, and variable-length input sequences. In this paper, we exploit the outstanding capability of path signature to translate online pen-tip trajectories into informative signature feature maps, successfully capturing the analytic and geometric properties of pen strokes with strong local invariance and robustness. A multi-spatial-context fully convolutional recurrent network (MC-FCRN) is proposed to exploit the multiple spatial contexts from the signature feature maps and generate a prediction sequence while completely avoiding the difficult segmentation problem. Furthermore, an implicit language model is developed to make predictions based on semantic context within a predicting feature sequence, providing a new perspective for incorporating lexicon constraints and prior knowledge about a certain language in the recognition procedure. Experiments on two standard benchmarks, Dataset-CASIA and Dataset-ICDAR, yielded outstanding results, with correct rates of 97.50 and 96.58 percent, respectively, which are significantly better than the best result reported thus far in the literature.

The development of convolutional neural networks (CNN) has led to revolutionary progress in the resolution of the offline handwritten Chinese character recognition (HCCR) problem. As the recognition rate on a standard offline HCCR testbed is outstanding, a few samples that remain misclassified have kindled our interest. In this paper, with the help of human recognition results, we present a comprehensive analysis of the samples misclassified by a state-of-the-art CNN model. We performed the analysis based on the top-1-votes, which are obtained from the statistical analysis of human recognition results, and derived the following conclusions: (1) the majority of samples with high top-1-votes were mis-labeled. Besides, by comparing the results of human recognition with that of CNN, some limitations of CNN that provide scope for further improvement are presented; (2) in the samples with medium top- 1-votes, it is shown that the samples with different confidence level have different characteristics. Specifically, some samples could be regarded as multi-label samples; (3) the samples with low top-1- votes are either wrongly written or written extensively in cursive style, which are difficult to match their given ground-truths; (4)the relationship between writing styles and misclassifications are also introduced in the paper. We believe this work should provide some insights and brings new clues on designing new classification methods to deal with these challenging samples.

Writer identification plays a significant role in several applications including forensic trace evidence identification, mobile bank transaction authentication, and handwritten character/text recognition. However, performing writer identification requires a considerable amount of experimental work and labor, as well as professional skills. In this paper, we propose a novel path-signature long short-term memory (PS-LSTM) recurrent neural network for writer identification that contributes as follows: 1) A mathematical feature set, path signature, is successfully applied to writer identification to characterize the essential geometric and analytic properties of pen-tip trajectory, which help distinguish between the diverse writing styles of different people, especially in confusing situations. 2) Different iterated integrals of path signature are investigated to model the local curvature nature of the pen-tip trajectory for writer identification. 3) Our study is the first to demonstrate the significance of integrating path signature with long short-term memory for writer identification. On Database CASIA-OLHWDN1.0, the proposed PS-LSTM writer identification achieves significantly superior results and outperforms previous works.

In this paper, we propose a new network architecture called Convolutional Multi-directional Recurrent Network (CDRN) for offline handwritten text recognition. The conventional recurrent neural network model obtains the local context from limited directions, whereas we build up the multi-directional long short-term memory (MDirLSTM) module to abstract contextual information in various directions. Moreover, we develop a shortcut connection strategy in our proposed architecture for faster yet better convergence. In cooperation with the aforementioned methods, the proposed architecture also benefits from the following properties: (1) it obtains informative features of the input directly without involving hand-crafted features and segmentation, and (2) it is an end-to-end trainable model whose components are trained conjointly. We evaluate the performance of the proposed method on two databases: IAM words and IRONOFF. Our experimental results demonstrate a significant increase in recognition performance using MDirLSTM and shortcut connections, which suggests the effectiveness of these two proposed methods.

This paper proposes an end-to-end framework, namely fully convolutional recurrent network (FCRN) for handwritten Chinese text recognition (HCTR). Unlike traditional methods that rely heavily on segmentation, our FCRN is trained with online text data directly and learns to associate the pen-tip trajectory with a sequence of characters. FCRN consists of four parts: a path-signature layer to extract signature features from the input pen-tip trajectory, a fully convolutional network to learn informative representation, a sequence modeling layer to make per-frame predictions on the input sequence and a transcription layer to translate the predictions into a label sequence. We also present a refined beam search method that efficiently integrates the language model to decode the FCRN and significantly improve the recognition results. We evaluate the performance of the proposed method on the test sets from the databases CASIA-OLHWDB and ICDAR 2013 Chinese handwriting recognition competition, and both achieve state-of-the-art performance with correct rates of 96.40% and 95.00%, respectively.

Deep convolutional neural networks (DCNNs) have achieved great success in various computer vision and pattern recognition applications, including those for handwritten Chinese character recognition (HCCR). However, most current DCNN-based HCCR approaches treat the handwritten sample simply as an image bitmap, ignoring some vital domain-specific information that may be useful but that cannot be learnt by traditional networks. In this paper, we propose an enhancement of the DCNN approach to online HCCR by incorporating a variety of domain-specific knowledge, including deformation, non-linear normalization, imaginary strokes, path signature, and 8-directional features. Our contribution is twofold. First, these domain-specific technologies are investigated and integrated with a DCNN to form a composite network to achieve improved performance. Second, the resulting DCNNs with diversity in their domain knowledge are combined using a hybrid serial-parallel (HSP) strategy. Consequently, we achieve a promising accuracy of 97.20% and 96.87% on CASIA-OLHWDB1.0 and CASIA-OLHWDB1.1, respectively, outperforming the best results previously reported in the literature.

Just like its great success in solving many computer vision problems, the convolutional neural networks (CNN) provided new end-to-end approach to handwritten Chinese character recognition (HCCR) with very promising results in recent years. However, previous CNNs so far proposed for HCCR were neither deep enough nor slim enough. We show in this paper that, a deeper architecture can benefit HCCR a lot to achieve higher performance, meanwhile can be designed with less parameters. We also show that the traditional feature extraction methods, such as Gabor or gradient feature maps, are still useful for enhancing the performance of CNN. We design a streamlined version of GoogLeNet [13], which was original proposed for image classification in recent years with very deep architecture, for HCCR (denoted as HCCR-GoogLeNet). The HCCR-GoogLeNet we used is 19 layers deep but involves with only 7.26 million parameters. Experiments were conducted using the ICDAR 2013 offline HCCR competition dataset. It has been shown that with the proper incorporation with traditional directional feature maps, the proposed single and ensemble HCCR-GoogLeNet models achieve new state of the art recognition accuracy of 96.35% and 96.74%, respectively, outperforming previous best result with significant gap.