Automating Information Extraction (IE) from handwritten documents is a challenging task due to the wide variety of handwriting styles, the presence of noise, and the lack of labeled data. In this work, we propose an end-toend encoder-decoder model, that incorporates transformers and Graph Convolutional Networks (GCN), to jointly perform Handwritten Text Recognition (HTR) and Named Entity Recognition (NER). The proposed architecture is mainly composed of two parts: a Sparse Graph Transformer Encoder (SGTE), to capture efficient representations of input text images while controlling the propagation of information through the model. The SGTE is followed by a transformer decoder enhanced with a GCN that combines the outputs of the last SGTE layer and the Multi-Head Attention (MHA) block to reinforce the alignment of visual features to characters and Named Entity (NE) tags, resulting in more robust learned representations. The proposed model shows promising results and achieves state-of-the-art performance on the IAM dataset, and in the ICDAR 2017 Information Extraction competition using the Esposalles database.

In this paper, we propose a Text-Degradation Invariant Auto Encoder (Text-DIAE), a self-supervised model designed to tackle two tasks, text recognition (handwritten or scene-text) and document image enhancement. We start by employing a transformer-based architecture that incorporates three pretext tasks as learning objectives to be optimized during pre-training without the usage of labelled data. Each of the pretext objectives is specifically tailored for the final downstream tasks. We conduct several ablation experiments that confirm the design choice of the selected pretext tasks. Importantly, the proposed model does not exhibit limitations of previous state-of-the-art methods based on contrastive losses, while at the same time requiring substantially fewer data samples to converge. Finally, we demonstrate that our method surpasses the state-of-the-art in existing supervised and self-supervised settings in handwritten and scene text recognition and document image enhancement. Our code and trained models will be made publicly available at https://github.com/dali92002/SSL-OCR

Document images can be affected by many degradation scenarios, which cause recognition and processing difficulties. In this age of digitization, it is important to denoise them for proper usage. To address this challenge, we present a new encoder-decoder architecture based on vision transformers to enhance both machine-printed and handwritten document images, in an end-to-end fashion. The encoder operates directly on the pixel patches with their positional information without the use of any convolutional layers, while the decoder reconstructs a clean image from the encoded patches. Conducted experiments show a superiority of the proposed model compared to the state-of-the-art methods on several DIBCO benchmarks. Code and models will be publicly available at: https://github.com/dali92002/DocEnTR.

Low resource Handwritten Text Recognition (HTR) is a hard problem due to the scarce annotated data and the very limited linguistic information (dictionaries and language models). For example, in the case of historical ciphered manuscripts, which are usually written with invented alphabets to hide the message contents. Thus, in this paper we address this problem through a data generation technique based on Bayesian Program Learning (BPL). Contrary to traditional generation approaches, which require a huge amount of annotated images, our method is able to generate human-like handwriting using only one sample of each symbol in the alphabet. After generating symbols, we create synthetic lines to train state-of-the-art HTR architectures in a segmentation free fashion. Quantitative and qualitative analyses were carried out and confirm the effectiveness of the proposed method.

Documents often exhibit various forms of degradation, which make it hard to be read and substantially deteriorate the performance of an OCR system. In this paper, we propose an effective end-to-end framework named document enhancement generative adversarial networks (DE-GAN) that uses the conditional GANs (cGANs) to restore severely degraded document images. To the best of our knowledge, this practice has not been studied within the context of generative adversarial deep networks. We demonstrate that, in different tasks (document clean up, binarization, deblurring and watermark removal), DE-GAN can produce an enhanced version of the degraded document with a high quality. In addition, our approach provides consistent improvements compared to state-of-the-art methods over the widely used DIBCO 2013, DIBCO 2017, and H-DIBCO 2018 datasets, proving its ability to restore a degraded document image to its ideal condition. The obtained results on a wide variety of degradation reveal the flexibility of the proposed model to be exploited in other document enhancement problems.

Handwritten text recognition in low resource scenarios, such as manuscripts with rare alphabets, is a challenging problem. In this paper, we propose a few-shot learning-based handwriting recognition approach that significantly reduces the human annotation process, by requiring only a few images of each alphabet symbols. The method consists of detecting all the symbols of a given alphabet in a textline image and decoding the obtained similarity scores to the final sequence of transcribed symbols. Our model is first pretrained on synthetic line images generated from an alphabet, which could differ from the alphabet of the target domain. A second training step is then applied to reduce the gap between the source and the target data. Since this retraining would require annotation of thousands of handwritten symbols together with their bounding boxes, we propose to avoid such human effort through an unsupervised progressive learning approach that automatically assigns pseudo-labels to the unlabeled data. The evaluation on different datasets shows that our model can lead to competitive results with a significant reduction in human effort. The code will be publicly available in the following repository: https://github.com/dali92002/HTRbyMatching

Vehicle re-identification (re-ID) aims to automatically find vehicle identity from a large number of vehicle images captured from multiple cameras. Most existing vehicle re-ID approaches rely on fully supervised learning methodologies, where large amounts of annotated training data are required, which is an expensive task. In this paper, we focus our interest on semi-supervised vehicle re-ID, where each identity has a single labeled and multiple unlabeled samples in the training. We propose a framework which gradually labels vehicle images taken from surveillance cameras. Our framework is based on a deep Convolutional Neural Network (CNN), which is progressively learned using a feature anchoring regularization process. The experiments conducted on various publicly available datasets demonstrate the efficiency of our framework in re-ID tasks. Our approach with only 20% labeled data shows interesting performance compared to the state-of-the-art supervised methods trained on fully labeled data.

Pan-sharpening is a fusion task, which aims to combine a low spatial resolution multispectral (MS) image with a high spatial resolution single band panchromatic (PAN) image to produce a high spatial and spectral Pan-sharpened image. The success of a Pan-sharpening technique depends on its ability to boost the spatial quality of the MS image while preserving its spectral feature. To this end, we propose in this article a new two-stage detail injection approach allowing to reconstruct fine structures based on convolutional neural networks (CNNs). First, generalized Laplacian pyramid gain injections CNN is performed to estimate the optimal values of the injection gains for each MS band to inject spatial details extracted from the PAN image. Next, the result is enhanced by injecting the details missing using the power of deep residual learning. The quantitative and qualitative results on data sets from different satellites show that the proposed approach can achieve higher performances in both spatial and spectral qualities compared to the state of the art as well as the new CNN-based methods.

Encoded (or ciphered) manuscripts are a special type of historical documents that contain encrypted text. The automatic recognition of this kind of documents is challenging because: 1) the cipher alphabet changes from one document to another, 2) there is a lack of annotated corpus for training and 3) touching symbols make the symbol segmentation difficult and complex. To overcome these difficulties, we propose a novel method for handwritten ciphers recognition based on few-shot object detection. Our method first detects all symbols of a given alphabet in a line image, and then a decoding step maps the symbol similarity scores to the final sequence of transcribed symbols. By training on synthetic data, we show that the proposed architecture is able to recognize handwritten ciphers with unseen alphabets. In addition, if few labeled pages with the same alphabet are used for fine tuning, our method surpasses existing unsupervised and supervised HTR methods for ciphers recognition.

In handwriting recognition, the design of relevant features is very important, but it is a daunting task. Deep neural networks are able to extract pertinent features automatically from the input image. This drops the dependency on handcrafted features, which is typically a trial and error process. In this paper, we perform an exhaustive experimental evaluation of learned against handcrafted features for Arabic handwriting recognition task. Moreover, we focus on the optimization of the competing full-word based language models by incorporating different characters and sub-words models. We extensively investigate the use of different sub-word-based language models, mainly characters, pseudo-words, morphemes and hybrid units in order to enhance the full-word handwriting recognition system for Arabic script. The proposed method allows the recognition of any out of vocabulary word as an arbitrary sequence of sub-word units. The KHATT database has been used as a benchmark for the Arabic handwriting recognition. We show that combining multiple language models enhances considerably the recognition performance for a morphologically rich language like Arabic. We achieve the state-of-the-art performance on the KHATT dataset.

Pan-sharpening aims to fuse a low-spatial-resolution multispectral (MS) image with an associated higher resolution panchromatic image (PAN) in order to produce a high-resolution MS (HRMS) image to overcome physical limitation of satellite sensors. In this letter, we propose a new generalized Laplacian pyramid gain injection prediction based on convolutional neural networks (GIP-CNN) for pan-sharpening, which estimates the values of the injection gains for each MS band to complement it with spatial details extracted from the PAN image. The experimental results on images from different satellites show that GIP-CNN can achieve higher performances with respect to the state-of-the-art and new CNN-based methods in both spatial and spectral qualities.

Person re-identification (re-ID) is an active task with several challenges such as variations of poses, view points, lighting and occlusion. When considering person re-ID as an image retrieval process, measuring the appearance similarity of a pairwise person images is the essential phase. Re-ranking process can improve its accuracy especially when it is based on an other similarity metric. In this paper, we propose a pipeline composed of two methods: A Siamese Convolutional Neural Network (S-CNN) and a k-reciprocal nearest neighbors (k-RNN) re-ranking algorithm. While most existing re-ranking methods ignore the importance of original distance in re-ranking, we jointly combine the S-CNN similarity measure and Jaccard distance to revise the initial ranked list. An experimental study is conducted on two benchmark person re-ID datasets (Market-1501 and Duke-MTMC-reID). The obtained results confirm the effectiveness of our method. A mAP improvement of 11.6% and 15.68% is obtained respectively for the two testing datasets.

This paper presents a deep learning approach for depressive episode detection on mobile devices, called DL4DED. It is based on a convolutional neural network and a long short-term memory network to identify the status of a patient’s voice extracted from spontaneous phone calls. To run DL4DED on mobile devices, two neural network model compression techniques are used: quantization and pruning. DL4DED protects data privacy, since it can be executed on a patient’s smartphone. Our proposal is validated on the DAIC-WOZ database. The obtained results show that the accuracy of DL4DED with model compression is only slightly lower than the accuracy of DL4DED without model compression. Furthermore, our experiments indicate that the power consumption of DL4DED is reasonably low.

We propose in this paper, an Arabic handwriting recognition system based on multiple BLSTM-CTC combination architectures. Given several feature sets, the low-level fusion consisted in projecting them into a unique feature space. Mid-level combination methods were performed using two techniques: the first one consists in averaging the a-posteriori probabilities of each individual BLSTM, and injecting them in the CTC decoding. The second is based on the training of a new BLSTM-CTC system using the sum of the a-posteriori probabilities generated by the individual systems. The high-level fusion is based on the combination of the individual decoding outputs. Lattice combination and ROVER strategies were evaluated in this context. The experiments conducted on the KHATT database showed that the high-level combination method significantly improves the recognition rate compared to the other fusion strategies.

Today's smartphones are able to capture documents with a good and simple way as any personal scanners. The captured document images need to be processed by specific and automated document processing systems. The systems are dedicated to textual content analysis, indexing and recognition. For instance, they may be used for font identification, writer identification and word or line segmentation. The state-of-the-art works lack comprehensive database for Arabic document images which are captured by mobile phones. This paper presents the first public offline images database for both printed and handwriting Arabic mobile captured documents, named "SmartATID". The document images of the database are acquired under varying capture conditions (blur, perspective angles and light). This causes photometric and geometric distortions that influence the performance of OCR process but also the page segmentation in lines and paragraphs. Each document image of our database is provided with a ground truth file that contains the exact text transcription and all numerical capture parameters used for each image capture. The database is freely and publicly usable by the research community at the following address http:// sites.google.com/site/smartatid.

The complexity and viariability of the Arabic handwriting makes difficult the implementation of an efficient recognition system through the use of a unique recognition engine. In this paper, two handwriting word recognition systems are combined in order to take advantage of their complementarities. The first one is a segmentation free based system that uses the generative classifier HMM. The second system is discriminative based. Relying on analytical approach, it proceeds with explicit segmentation of words into graphemes. Different combination strategies are compared including sum, product, Borda count and Dempster-Shafer rules. The experimental results conducted on ADAB database demonstrate a significant improvement of recognition accuracy of 5% compared to the segmentation free based system and 9% compared to the analytical based system.

In this paper, we propose a novel system for keyword spotting in handwritten documents. Our approach proceeds in two steps: first a generic text line HMM provides a simple and flexible tool to localize the keyword and its character boundaries. In the second step, a SVM based verification system estimates and combines the character probabilities to provide keyword confidence scores which are further combined with the HMM score. The system has been evaluated on a public handwritten document database used for the 2011 ICDAR handwriting recognition competitions and shows that the verification stage improves the performance and outperforms some other state-of-the-art approaches.

In this paper, we propose a novel system for word spotting and regular expression detection in Handwritten documents. The proposed approach is lexicon-free, i.e., able to spot arbitrary keywords that are not required to be known at the training stage. Furthermore, the proposed system is segmentation-free, i.e., text lines are not required to be segmented into words. The originalities of our approach is twofold. First we propose a new filler model which allows to speed-up the decoding process. Second, we extend the methodology to search for regular expressions. The system has been evaluated on a public handwritten document database used for the 2011 ICDAR handwriting recognition competitions.

In this paper, a novel rejection strategy is proposed to optimize the reliability of an handwritten word recognition system. The proposed approach is based on several steps. First, we combine the outputs of several HMM classifiers using the Dempster-Shafer theory (DST). Then, we take advantage of the expressivity of mass functions (the counter part of probability distributions in DST) to characterize the quality/reliability of the classification. Finally, we use this characterization to decide whether a test word is rejected or not. Experiments carried out on RIMES and IFN/ENIT datasets show that the proposed approach outperforms other state-of-the-art rejection methods.

This paper is focused on the optimization of the computational efficiency of a multi-stream word recognition system. The aim of this work is to optimize the multi-stream decoding step in order to reduce the recognition time and the complexity to allow combining a large number of streams. Two different multi-stream decoding strategies are compared based on two-level and HMM-recombination algorithms. Experiments carried out on public handwritten word databases show significant speed gains at decoding while keeping the same performances, in addition to new insights for combining a large number of streams.

In this work, we focus on an improvement of a road obstacle recognition system using SVM based classifiers combination. The improvement relies on the use of Dempster-Shafer theory (DST) to combine in a finer way the outputs of SVM classifiers. The SVM classifiers were trained on different local and global features based on Speeded Up Robust Features (SURF) extracted from both visible and far-infrared images. A two-stage recognition method is also proposed to reduce the complexity of the overall system. The experiments are conducted on a set of images where obstacles occur at different scales, shapes and in difficult recognition situations. They show significant improvements while using DST combination compared to the classical combination strategies.

The classification process in handwriting recognition is designed to provide lists of results rather than single results, so that context models can be used as post-processing. Most of the time, the length of the list is determined once and for all the items to classify. Here, we present a method based on Dempster-Shafer theory that allows a different length list for each item, depending on the precision of the information involved in the decision process. As it is difficult to compare the results of such an algorithm to classical accuracy rates, we also propose a generic evaluation methodology. Finally, this algorithm is evaluated on Latin and Arabic handwritten isolated word datasets.

Generally, handwritten word recognition systems use script specific methodologies. In this paper, we present a unified approach for multi-lingual recognition of alphabetic scripts. The proposed system operates independently of the nature of the script using the multi-stream paradigm. The experiments have been carried out on a multi-script database composed of Arabic and Latin handwritten words from the IFN/ENIT and the IRONOFF public databases and show interesting recognition performances with only 1.5% of script confusion and an overall word recognition rate of 84.5% using a multi-script lexicon of 1142 words.

The multi-stream paradigm provides an interesting framework for the integration of multiple sources of information. In this paper, we present our multi-script recognition system using the multi-stream formalism to combine low level feature streams. We Analyze how the combination of n streams (n=2,....,4) can improve the recognition performance. Significant experiments have been carried out on two publicly available word databases: IFN/ENIT benchmark database (Arabic script) and IRONOFF database (Latin script). The proposed framework shows interesting results in both cases thanks to the use of low level features in a segmentation free approach which ensure its applicability to various scripts.

We present in this paper a new approach based on multi-stream hidden Markov models (HMM) for the recognition of off-line handwriting. Every word is presented by two HMM models: the first one is learned with features extracted from upper contour, the second with features extracted from lower contour. The combination of these two sources of information is studied using the multi-stream framework. We present experiment results obtained on a database composed of isolated words extracted from incoming mail documents.

In this paper we present a method based on hidden Markov random fields and 2D dynamic programming image decoding, for segmenting pages of complex handwritten manuscripts such as novelist drafts. After a formal description of the theoretical framework and the principles of the decoding method, we describe the implementation of the model and the decoding method. Then we discuss the results obtained with this approach on the drafts of the French novelist Gustave Flaubert.