Transformer-based encoder-decoder networks have recently achieved impressive results in handwritten text recognition, partly thanks to their auto-regressive decoder which implicitly learns a language model. However, such networks suffer from a large performance drop when evaluated on a target corpus whose language distribution is shifted from the source text seen during training. To retain recognition accuracy despite this language shift, we propose an external n-gram injection (NGI) for dynamic adaptation of the network's language modeling at inference time. Our method allows switching to an n-gram language model estimated on a corpus close to the target distribution, therefore mitigating bias without any extra training on target image-text pairs. We opt for an early injection of the n-gram into the transformer decoder so that the network learns to fully leverage text-only data at the low additional cost of n-gram inference. Experiments on three handwritten datasets demonstrate that the proposed NGI significantly reduces the performance gap between source and target corpora.

The recognition of mid-18th to mid-20th century piano scores presents segmentation challenges caused by touching and broken symbols produced by imprinting techniques and time degradation. We present a new notehead accidental dataset containing 2955 images from dense and damaged piano scores. We address this detection problem with very small training samples using a simple Spatial Transformer (ST)-based Convolutional Neural Network detector improved through bootstrapping and contextual information, and more powerful deep learning detectors (Faster R-CNN, R-FCN, and SSD) with transfer-learning on the COCO dataset. We trained all our detectors using 5 fold cross-validation and obtain 98.73% mean Average Precision (mAP) for an Intersection over Union (IoU) threshold of 0.75 with our best detector. Our ST-based detector obtains a slightly lower mAP of 94.81%, but runs 40 times faster, and uses 18 times less memory.

In this article, we present our work on baseline detection in images of historical documents. This work focuses on handwritten documents containing tabular structures. One of the difficulties of this kind of documents is the strong interaction between text and tabular structures. This interaction leads to ambiguous cases for which recognition systems often over-or sub-segment baselines. The interest of our method is to combine contextual and structural knowledge in order to interpret properly this interaction. Our combination is able to merge heterogeneous information obtained with a deep-learning approach (for contextual elements) and a syntactical approach (for structural elements). Our grammatical description consists on a logical description of the intersections between text-lines and vertical rulings of detected tables. Intersections are described thanks to physical indicators extracted from images: vertical rulings, hypothetical text-lines, begin-and end-indicators of text-lines. We show on cBAD competition [4] (competition on baseline detection) that the combination of heterogeneous knowledge (structural and contextual information) improves baseline detection in handwritten documents. We obtain better scores than the best method published until now on this competition.

In this paper, we present an evaluative study of pixel-labeling methods using the HBA 1.0 dataset for historical book analysis. This study is held in the context of the 2nd historical book analysis (HBA2019) competition and in conjunction with the 15th IAPR international conference on document analysis and recognition (ICDAR2019). The HBA2019 competition provides a large experimental corpus and a thorough evaluation protocol to ensure an objective performance benchmarking of pixel-labeling document image methods. Two nested challenges are evaluated in the HBA2019 competition: Challenge 1 and Challenge 2. Challenge 1 evaluates how image analysis methods could discriminate the textual content from the graphical ones at pixel level. Challenge 2 assesses the capabilities of pixel-labeling methods to separate the textual content according to different text fonts (e.g. lowercase, uppercase, italic, etc.) at pixel level. During the competition, we received 52 and 38 different teams' registrations for Challenge 1 and Challenge 2, respectively and finally 5 of them submitted their results in each challenge. Qualitative and numerical results of the participating methods in both challenges are reported and discussed in this paper in order to provide a baseline for future evaluation studies in historical document image analysis. The evaluation shows that the method submitted by the NLPR-CASIA team achieves the highest performance in both challenges.

Optical Music Recognition (OMR) is the challenge of understanding the content of musical scores. Accurate detection of individual music objects is a critical step in processing musical documents because a failure at this stage corrupts any further processing. So far, all proposed methods were either limited to typeset music scores or were built to detect only a subset of the available classes of music symbols. In this work, we propose an end-to-end trainable object detector for music symbols that is capable of detecting almost the full vocabulary of modern music notation in handwritten music scores. By training deep convolutional neural networks on the recently released MUSCIMA++ dataset which has symbol-level annotations, we show that a machine learning approach can be used to accurately detect music objects with a mean average precision of over 80%.

Dealing with non annotated documents for the design of a document recognition system is not an easy task. In general, statistical methods cannot learn without an annotated ground truth, unlike syntactical methods. However their ability to deal with non annotated data comes from the fact that the description is manually made by a user. The adaptation to a new kind of document is then tedious as the whole manual process of extraction of knowledge has to be redone. In this paper, we propose a method to extract knowledge and generate rules without any ground truth. Using large volume of non annotated documents, it is possible to study redundancies of some extracted elements in the document images. The redundancy is exploited through an automatic clustering algorithm. An interaction with the user brings semantic to the detected clusters. In this work, the extracted elements are some keywords extracted with word spotting. This approach has been applied to old marriage record field detection on the FamilySearch HIP2013 competition database. The results demonstrate that we successfully automatically infer rules from non annotated documents using the redundancy of extracted elements of the documents.

A critical step of on-line handwritten diagram recognition is the segmentation between text and symbols. It is still an open problem in several approaches of the literature. However, for a human operator, text/symbol segmentation is an easy task and does not even need understanding diagram semantics. It is done thanks to the use of both structural knowledge and statistical analysis. A human operator knows what is a symbol and how to distinguish a good symbol from a bad one in a list of candidates. We propose to reproduce this perceptive mechanism by introducing some statistical information inside of a grammatical method for document structure recognition, in order to combine both structural an statistical knowledge. This approach is applied to flowchart recognition on a freely available database. The results demonstrate the interest of combining statistical and structural information for perceptive vision in diagram recognition.

Extracting and indexing meaningful contents from degraded documents, like historical ones, is a challenging problem. Existing analysis systems usually rely on a manual correction of results during the post-processing stage, and cannot make use of external information to adapt their response. This paper presents how an existing document analysis system can be easily adapted to enable an efficient interaction during the analysis stage, and benefit from external information. We identify the minimal architecture required, and we detail the two complimentary interaction models we propose: a directed interaction model which allows to handle cases where errors can be automatically detected, and a spontaneous interaction model which permits to cope with the other cases. Both models are asynchronous to prevent the human operator or the system from waiting for each other during document processing. They are based on a common foundation which uses standard exception-like mechanisms to implement error detection, correction and recovery aspects. Our system was tested on several tasks. For instance, for the transcription of handwritten words in documents dating from the 18th century, where we were able to diminish the human workload by 28% for an overall recognition rate of 80%.

The analysis of sets of degraded documents, like historical ones, is error-prone and requires human help to achieve acceptable quality levels. However, human interaction raises 3 main issues when processing important amounts of pages: none of the user or the system should wait for work, information provided by a human operator should not be restricted to local isolated corrections, but rather produce durable changes in the system, the ability to interact with a human operator should not increase the complexity of document models nor duplicate them between analysis and human interaction processes. To solve those issues, we propose an iterative approach, based on a special mechanism called visual memory, to reintegrate external information during page analysis. So as to demonstrate the interest for existing systems, we explain how we adapted a (rule-based) page analysis tool to enable, in this iterative approach, a delayed interaction with a human operator based on an adaptation of error recovery principles for compilers and the well-known exception handling mechanism. We validated our iterative approach on sales registers from the 18th century.

Transcription of handwritten words in historical documents is still a difficult task. When processing huge amount of pages, document-centered approaches are limited by the trade-off between automatic recognition errors and the tedious aspect of human user annotation work. In this article, we investigate the use of inter page dependencies to overcome those limitations. For this, we propose a new architecture that allows the exploitation of handwritten word redundancies over pages by considering documents from a higher point of view, namely the collection level. The experiments we conducted on handwritten word transcription show promising results in terms of recognition error and human user work reductions.

When reading a document, we intuitively have a first global approach in order to determine the whole structure, before reading parts in details. We propose to apply the same kind of mechanism by introducing the concept of multiresolution in an existing generic method for structured document recognition. This new combination of different vision levels makes it possible to recognize low structured documents. We present our work on an example: the multiresolution description of archive documents that are naturalization decree registers from the 19th and 20th century. The validation has been made on 85,088 images. Integrated in a platform for archive documents, the located elements offers to users a fast leaf through naturalization decrees.

In this paper, we propose to give tools for Bangla handwriting recognition. We present a mechanism to segment documents into text lines and words, and more specifically to detect headline position in each word. Indeed, this headline is an horizontal line on the upper part of most of characters, which is characteristic of Bangla writing. Its localisation is a new approach that can improve text recognition quality. This headline is detected into words inside text lines thanks to a notion of perceptive vision: at a certain distance, text lines appear as line-segments that give the global orientation of words. Watching closer may help to give the exact position of the headline. Consequently, this work is mainly based on applying a segment extractor at different image resolutions and combining extracted information in order to compute the headlines. Our line-segment extractor is based on Kalman filtering.

This paper presents annotations needed for handwritten archive document retrieval by content. We propose two complementary ways of producing those annotations: automatically by using optical document recognition and collectively by using the Internet and manual input by users. A platform for managing those annotations is presented as well as examples of automatic annotations on civil status registers, military forms (tested on 60,000 pages) and naturalization decrees, using a generic document recognition method. Examples of collective annotations built on automatic annotations are also given.

Genericity in structured document recognition is a difficult challenge. We therefore propose a new generic document recognition method, called DMOS (Description and MOdification of Segmentation), that is made up of a new grammatical formalism, called EPF (Enhanced Position Formalism) and an associated parser which is able to introduce context in segmentation. We implement this method to obtain a generator of document recognition systems. This generator can automatically produce new recognition systems. It is only necessary to describe the document with an EPF grammar, which is then simply compiled. In this way, we have developed various recognition systems: one on musical scores, one on mathematical formulae and one on recursive table structures. We have also defined a specific application to damaged military forms of the 19th Century. We have been able to test the generated system on 5,000 of these military forms. This has permitted us to validate the DMOS method on a real-world application.

In this paper we present a real-world evaluation of DMOS, a new generic document recognition method. This method uses a new grammatical formalism (EPF) and an associated parser able to introduce context in segmentation. We have implemented this DMOS method to build an automatic generator of structured document recognition systems. We already produced three recognition systems by only changing the EPF grammar: one on musical scores, one on mathematical formulae and one on recursive table structures. We present here a specific light grammar to automatically recognize quite damaged 19th century military forms. The quality of those forms is far from perfect: table lines are not well printed, paper is so thin that there are transparency problems (the forms are two-sided) but the biggest problem comes from small paper sheets hiding part of the structure. The evaluation of this system has been made onto 5268 images and the results show that the system did not make any mistake. Moreover it can recognize the entire structure in 97.2% of the forms (the other 2.8% are automatically set apart).

Optical Music Recognition is a form of document analysis for which a priori knowledge is particularly important. Musical notation is governed by a substantial set of rules, but current systems fail to use them adequately. In complex scores, existing systems cannot overcome the well-known segmentation problems of document analysis, due mainly to the high density of music information. This paper proposes a new method of recognition which uses a grammar in order to formalize the syntactic rules and represent the context. However, where objects touch, there is a discrepancy between the way the existing knowledge (grammar) will describe an object and the way it is recognized, since touching objects have to be segmented first. Following a description of the grammar, this paper shall go on to propose the use of an operator to modify the way the grammar parses the image so that the system can deal with certain touching objects (e.g. where an accidental touches a notehead).