Depression is a major contributor to the global disease burden and is closely associated with variations in facial expressions, making them a critical biomarker for its recognition. Traditional local binary convolutional neural networks (LBCNN) rely on fixed binary weights, which fail to capture complex and nuanced features, limiting their adaptability and generalization. Existing methods often lack extracting spatial hierarchies and model long-range dependencies within facial expressions. To address these gaps, an enhanced random binary multilevel attention network (ERBMA-Net) is proposed for facial depression recognition, which employs a multibranch architecture to extract global (face) and local (eyes and mouth) features. Subsequently, the enhanced random binary convolutional neural network (ERBCNN) introduces random binary filters and a refinement layer to improve adaptability over LBCNN. Furthermore, multilevel attention mechanisms, including spatial attention are applied independently to global and local features and self-attention on concatenated features to capture critical spatial and relational patterns. Comprehensive evaluations on the Audio-Visual Emotion Challenge 2014 (AVEC2014) and the Changzhou No. 2 People’s Hospital (CZ2024) datasets demonstrate the superior performance of ERBMA-Net, achieving mean absolute error and root mean square error: 6.80/8.18 and 6.37/8.08, respectively. These results establish ERBMA-Net as a robust framework for artificial intelligence-assisted mental health diagnostics. The code is available at:https://github.com/DrTuryalai/ERBMA-Net.

Abstract Deploying state-of-the-art document understanding models remains resource-intensive and impractical in many real-world scenarios, particularly where labeled data is scarce and computational budgets are constrained. To address these challenges, this work proposes a novel approach towards parameter-efficient document understanding models capable of adapting to specific tasks and document types without the need for labeled data. Specifically, we propose an approach coined SlimDoc to distill multimodal document transformer encoder models into smaller student models, using internal signals at different training stages, followed by external signals. Our approach is inspired by TinyBERT and adapted to the domain of document understanding transformers. We demonstrate SlimDoc to outperform both a single-stage distillation and a direct fine-tuning of the student. Experimental results across six document understanding datasets demonstrate our approach’s effectiveness: Our distilled student models achieve on average $$93.0\%$$ 93.0 % of the teacher’s performance, while the fine-tuned students achieve $$87.0\%$$ 87.0 % of the teacher’s performance. Without requiring any labeled data, we create a compact student which achieves $$96.0\%$$ 96.0 % of the performance of its supervised-distilled counterpart and $$86.2\%$$ 86.2 % of the performance of a supervised-fine-tuned teacher model. We demonstrate our distillation approach to pick up on document geometry and to be effective on the two popular document understanding models LiLT and LayoutLMv3. Our implementation and training data is available at https://github.com/marcel-lamott/SlimDoc .

We present a novel extension to embedding-based knowledge graph completion models which enables them to perform open-world link prediction, i.e. to predict facts for entities unseen in training based on their textual description. Our model combines a regular link prediction model learned from a knowledge graph with word embeddings learned from a textual corpus. After training both independently, we learn a transformation to map the embeddings of an entity’s name and description to the graph-based embedding space.In experiments on several datasets including FB20k, DBPedia50k and our new dataset FB15k-237-OWE, we demonstrate competitive results. Particularly, our approach exploits the full knowledge graph structure even when textual descriptions are scarce, does not require a joint training on graph and text, and can be applied to any embedding-based link prediction model, such as TransE, ComplEx and DistMult.

Hyperspectral Document Image (HSDI) analysis allows for efficient and accurate differentiation of inks with visually similar color but unique spectral response, which is a crucial step in authentication of documents. Various HSDI based ink discrimination methods are available in the current literature, however, more accurate and robust methods are required to empower document authentication. Contrary to the former ink mismatch detection methods based on spectral features only, we present a novel method based on deep learning that exploits the spectral correlation as well as the spatial context to enhance ink mismatch detection. Spectral responses of the target pixel and its neighboring pixels are organized in an image format and fed to a Convolutional Neural Network (CNN) for classification. The proposed method achieves the highest accuracy among the other ink mismatch detection methods on the UWA Writing Ink Hyperspectral Images database (WIHSI), which demonstrates the effectiveness of deep learning models employing spatio-spectral hybrid features for document authentication. Detailed experimental analysis for selection of appropriate CNN architecture, spatio-spectral data format and training ratio is presented along with a comparison with the previous methods on this subject.

Tables present summarized and structured information to the reader, which makes table's structure extraction an important part of document understanding applications. However, table structure identification is a hard problem not only because of the large variation in the table layouts and styles, but also owing to the variations in the page layouts and the noise contamination levels. A lot of research has been done to identify table structure, most of which is based on applying heuristics with the aid of optical character recognition (OCR) to hand pick layout features of the tables. These methods fail to generalize well because of the variations in the table layouts and the errors generated by OCR. In this paper, we have proposed a robust deep learning based approach to extract rows and columns from a detected table in document images with a high precision. In the proposed solution, the table images are first pre-processed and then fed to a bi-directional Recurrent Neural Network with Gated Recurrent Units (GRU) followed by a fully-connected layer with softmax activation. The network scans the images from top-to-bottom as well as left-to-right and classifies each input as either a row-separator or a column-separator. We have benchmarked our system on publicly available UNLV as well as ICDAR 2013 datasets on which it outperformed the state-of-theart table structure extraction systems by a significant margin.

Document structure analysis, such as zone segmentation and table recognition, is a complex problem in document processing and is an active area of research. The recent success of deep learning in solving various computer vision and machine learning problems has not been reflected in document structure analysis since conventional neural networks are not well suited to the input structure of the problem. In this paper, we propose an architecture based on graph networks as a better alternative to standard neural networks for table recognition. We argue that graph networks are a more natural choice for these problems, and explore two gradient-based graph neural networks. Our proposed architecture combines the benefits of convolutional neural networks for visual feature extraction and graph networks for dealing with the problem structure. We empirically demonstrate that our method outperforms the baseline by a significant margin. In addition, we identify the lack of large scale datasets as a major hindrance for deep learning research for structure analysis and present a new large scale synthetic dataset for the problem of table recognition. Finally, we open-source our implementation of dataset generation and the training framework of our graph networks to promote reproducible research in this direction.

Multispectral imaging allows for analysis of images in multiple spectral bands. Over the past three decades, airborne and satellite multispectral imaging have been the focus of extensive research in remote sensing. In the recent years, ground based multispectral imaging has gained an immense amount of interest in the fields ranging from computer vision and medical imaging to art, archaeology and computational forensics. The rich information content in multispectral images allows forensic experts to examine the chemical composition of forensic traces. Due to its rapid, non-contact and non-destructive characteristics, multispectral imaging is an effective tool for visualization, age estimation, detection and identification of forensic traces in document images. Ink mismatch is a key indicator of forgery in a document. Inks of different materials exhibit different spectral signature even if they have the same color. Multispectral analysis of questioned documents images allows identification and discrimination of visually similar inks. In this paper, an efficient automatic ink mismatch detection technique is proposed which uses Fuzzy C-Means Clustering to divide the spectral responses of ink pixels in handwritten notes into different clusters which relate to the unique inks used in the document. Sauvola's local thresholding technique is employed to efficiently segment foreground text from the document image. Furthermore, feature selection is used to optimize the performance of the proposed method. The presented method provides better ink discrimination results than state-of-the-art methods when tested on publicly available UWA Writing Inks Dataset.

Content based information search and retrieval has allowed for easier access to data. While Latin based scripts have gained attention and support from academia and industry, there is limited support for cursive script languages, like Urdu. In this paper, we present the first instance of Urdu news ticker detection and recognition and take a micron sized step towards the goal of super intelligence. The presented solution allows for automating the transcription, indexing and captioning of Urdu news video content. We present the first comprehensive data set, to our knowledge, for Urdu news ticker recognition, collected from 41 different news channels. The data set covers both high and low quality channels, distorted and blurred news tickers, making the data set an ideal test case for any automatic Urdu News Recognition system in future. We identify and address the key challenges in Urdu News Ticker text recognition. We further propose an adjustment to the ground-truth labeling strategy focused on improving the readability of recognized output. Finally, we propose and present results from a Bi-Directional Long Short-Term Memory (BDLSTM) network architecture for news ticker text recognition. Our custom trained model outperforms Google's commercial OCR engine in two of the four experiments conducted.

We present an algorithm that automatically establishes dense correspondences between a large number of 3D faces. Starting from automatically detected sparse correspondences on the outer boundary of 3D faces, the algorithm triangulates existing correspondences and expands them iteratively by matching points of distinctive surface curvature along the triangle edges. After exhausting keypoint matches, further correspondences are established by generating evenly distributed points within triangles by evolving level set geodesic curves from the centroids of large triangles. A deformable model (K3DM) is constructed from the dense corresponded faces and an algorithm is proposed for morphing the K3DM to fit unseen faces. This algorithm iterates between rigid alignment of an unseen face followed by regularized morphing of the deformable model. We have extensively evaluated the proposed algorithms on synthetic data and real 3D faces from the FRGCv2, Bosphorus, BU3DFE and UND Ear databases using quantitative and qualitative benchmarks. Our algorithm achieved dense correspondences with a mean localisation error of 1.28 mm on synthetic faces and detected 14 anthropometric landmarks on unseen real faces from the FRGCv2 database with 3 mm precision. Furthermore, our deformable model fitting algorithm achieved 98.5 percent face recognition accuracy on the FRGCv2 and 98.6 percent on Bosphorus database. Our dense model is also able to generalize to unseen datasets.

First-person videos have unique characteristics such as heavy egocentric motion, strong preceding events, salient transitional activities and post-event impacts. Action recognition methods designed for third person videos may not optimally represent actions captured by first-person videos. We propose a method to represent the high level dynamics of sub-events in first-person videos by dynamically pooling features of sub-intervals of time series using a temporal feature pooling function. The sub-event dynamics are then temporally aligned to make a new series. To keep track of how the sub-event dynamics evolve over time, we recursively employ the Fast Fourier Transform on a pyramidal temporal structure. The Fourier coefficients of the segment define the overall video representation. We perform experiments on two existing benchmark first-person video datasets which have been captured in a controlled environment. Addressing this gap, we introduce a new dataset collected from YouTube which has a larger number of classes and a greater diversity of capture conditions thereby more closely depicting real-world challenges in first-person video analysis. We compare our method to state-of-the-art first person and generic video recognition algorithms. Our method consistently outperforms the nearest competitors by 10.3%, 3.3% and 11.7% respectively on the three datasets.

Hyperspectral imaging helps in identifying patterns and objects in an observed hyperspectral scene on the basis of their unique spectral signatures; such identification is otherwise difficult using regular imaging. Recently, ink mismatch detection analysis based on hyperspectral imaging has shown enormous potential in distinguishing visually similar inks. Such analysis provides significant information to forensic document examiners to determine the authenticity of the questioned documents. However, a major challenge still exists in disproportionate ink mismatch detection because it is inherently an unbalanced clustering problem. The presented approach deals with ink mismatch detection in unbalanced clusters by using hyperspectral unmixing scheme. It identifies the spectral signatures (endmembers) of the inks and their corresponding proportions (abundances). Our results show that HySime outperforms other methods in signal subspace estimation. Hyperspectral unmixing is done by using minimum volume enclosing simplex algorithm. Efficacy of the purposed approach is demonstrated by successfully distinguishing varying disproportionate ink datasets generated from UWA database and results are compared with existing state of the art methods in hyperspectral ink mismatch detection field. We expect that these finding will further encourage the use of hyperspectral imaging in document analysis, particularly towards automated questioned document examination.

Table detection is a crucial step in many document analysis applications as tables are used for presenting essential information to the reader in a structured manner. It is a hard problem due to varying layouts and encodings of the tables. Researchers have proposed numerous techniques for table detection based on layout analysis of documents. Most of these techniques fail to generalize because they rely on hand engineered features which are not robust to layout variations. In this paper, we have presented a deep learning based method for table detection. In the proposed method, document images are first pre-processed. These images are then fed to a Region Proposal Network followed by a fully connected neural network for table detection. The proposed method works with high precision on document images with varying layouts that include documents, research papers, and magazines. We have done our evaluations on publicly available UNLV dataset where it beats Tesseract's state of the art table detection system by a significant margin.

Seamless integration of information from digital and paper documents is crucial for efficient knowledge management. One convenient way to achieve this is to digitize a document from a natural image. This requires precise localization of the document in the image. Several methods have been proposed to solve this problem but they rely on traditional image processing techniques which are not robust to extreme viewpoint and background variations. Deep Convolutional Neural Networks (CNNs), on the other hand, have shown to be extremely robust to variations in background and viewpoint in object detection and classification tasks. Inspired by their robustness and generality, we propose a novel CNN based method to accurately localize documents in real-time. We model localization problem as a key point detection problem. The four corners of the documents are jointly predicted by a Deep Convolutional Neural Network. We then refine our prediction using a novel recursive application of a CNN. Performance of the system is evaluated on ICDAR 2015 SmartDoc Competition 1 dataset. The results are comparable to state of the art on simple backgrounds and improve the state of the art to 94% from the previous 86% on the complex background. Code, dataset, and models are available at: https://github.com/KhurramJaved96/Recursive-CNNs.

A major hurdle in the development of practical Urdu Nastaleeq script OCR is the lack of transcribed data, which is a pre-requisite for training machine learning algorithms. Most of the previous research has focused on UPTI, a publicly available data set with no particular focus on performance on real world images. UPTI contains only 6000 of the most probable 26,000 ligatures of Urdu. We build upon UPTI with a new data set, UPTI 2.0 that covers over 18,000 ligatures of Urdu Nastaleeq, hence covering over 70% of the ligatures that can practically occur. We further train a system on UPTI 2.0 and compare its performance against the only commercial Urdu Nastaleeq OCR system to date. Bidirectional Long Short-Term Memory (BDLSTM) network is employed with Connectionist Temporal Classification (CTC) layer as the recognizer. We show that systems trained on UPTI 2.0 outperform the commercial system.

B This paper presents a novel approach, named D-StaR, for stamp segmentation from scanned document images. The presented approach is generic (applicable to stamps of any color, shape, size, and orientation) and based on deep learning. In particular, it uses Fully Convolutional networks for semantic analysis of documents to extract stamps. The presented approach is evaluated on a publicly available stamp dataset. Evaluation results show that the presented approach outperforms the state-of-the-art methods for stamp segmentation and achieves pixel based precision and recall of 87% and 84%, respectively. Deeper analysis of the evaluation reveals that the presented approach can segment both overlapping and non-overlapping stamps, which was always a problem for existing systems in the literature.

Documents are routinely captured by digital cameras in today's age owing to the availability of high quality cameras in smart phones. However, recognition of camera-captured documents is substantially more challenging as compared to traditional flat bed scanned documents due to the distortions introduced by the cameras. One of the major performancelimiting artifacts is the motion and out-of-focus blur that is often induced in the document during the capturing process. Existing approaches try to detect presence of blur in the document to inform the user for re-capturing the image. This paper reports, for the first time, an Optical Character Recognition (OCR) system that can directly recognize blurred documents on which the stateof-the-art OCR systems are unable to provide usable results. Our presented system is based on the Long Short-Term Memory (LSTM) networks and has shown promising character recognition results on both the motion-blurred and out-of-focus blurred images. One important feature of this work is that the LSTM networks have been applied directly to the gray-scale document images to avoid error-prone binarization of blurred documents. Experiments are conducted on publicly available SmartDoc-QA dataset that contains a wide variety of image blur degradations. Our presented system achieves 12.3% character error rate on the test documents, which is an over three-fold reduction in the error rate (38.9%) of the best-performing contemporary OCR system (ABBYY Fine Reader) on the same data.

In this paper, we propose a method for automatic signature segmentation using hyper-spectral imaging. The proposed method first uses the connected component analysis and local features to segment the printed text and signatures. Secondly, it uses spectral response of text, signature, and background to extract signature pixels. The proposed method is robust, and remains unaffected by color and intensity of the ink, and by any structural information of the text, as the classification relies exclusively on the spectral response of the document. The proposed method can extract signature pixels either overlapping or non-overlapping from different backgrounds like, logos, tables, stamps, and printed text. We used high-resolution hyper-spectral imaging to study and classify 300 documents with varying backgrounds. We evaluated the proposed classification method and compared results with the state-of-the art system. The proposed method outperformed the state-of-the-art system and achieved 100% precision and 84% recall.

Deep learning based methods have achieved unprecedented success in solving several computer vision problems involving RGB images. However, this level of success is yet to be seen on RGB-D images owing to two major challenges in this domain: training data deficiency and multi-modality input dissimilarity. We present an RGB-D object recognition framework that addresses these two key challenges by effectively embedding depth and point cloud data into the RGB domain. We employ a convolutional neural network (CNN) pre-trained on RGB data as a feature extractor for both color and depth channels and propose a rich coarse-to-fine feature representation scheme, coined Hypercube Pyramid, that is able to capture discriminatory information at different levels of detail. Finally, we present a novel fusion scheme to combine the Hypercube Pyramid features with the activations of fully connected neurons to construct a compact representation prior to classification. By employing Extreme Learning Machines (ELM) as non-linear classifiers, we show that the proposed method outperforms ten state-of-the-art algorithms for several tasks in terms of recognition accuracy on the benchmark Washington RGB-D and 2D3D object datasets by a large margin (upto 50% reduction in error rate).

We propose a Bayesian approach to learn discriminative dictionaries for sparse representation of data. The proposed approach infers probability distributions over the atoms of a discriminative dictionary using a finite approximation of Beta Process. It also computes sets of Bernoulli distributions that associate class labels to the learned dictionary atoms. This association signifies the selection probabilities of the dictionary atoms in the expansion of class-specific data. Furthermore, the non-parametric character of the proposed approach allows it to infer the correct size of the dictionary. We exploit the aforementioned Bernoulli distributions in separately learning a linear classifier. The classifier uses the same hierarchical Bayesian model as the dictionary, which we present along the analytical inference solution for Gibbs sampling. For classification, a test instance is first sparsely encoded over the learned dictionary and the codes are fed to the classifier. We performed experiments for face and action recognition; and object and scene-category classification using five public datasets and compared the results with state-of-the-art discriminative sparse representation approaches. Experiments show that the proposed Bayesian approach consistently outperforms the existing approaches.

Despite the proven efficacy of hyperspectral imaging in many computer vision tasks, its widespread use is hindered by its low spatial resolution, resulting from hardware limitations. We propose a hyperspectral image super resolution approach that fuses a high resolution image with the low resolution hyperspectral image using non-parametric Bayesian sparse representation. The proposed approach first infers probability distributions for the material spectra in the scene and their proportions. The distributions are then used to compute sparse codes of the high resolution image. To that end, we propose a generic Bayesian sparse coding strategy to be used with Bayesian dictionaries learned with the Beta process. We theoretically analyze the proposed strategy for its accurate performance. The computed codes are used with the estimated scene spectra to construct the super resolution hyperspectral image. Exhaustive experiments on two public databases of ground based hyperspectral images and a remotely sensed image show that the proposed approach outperforms the existing state of the art.

We present an algorithm for automatic detection of a large number of anthropometric landmarks on 3D faces. Our approach does not use texture and is completely shape based in order to detect landmarks that are morphologically significant. The proposed algorithm evolves level set curves with adaptive geometric speed functions to automatically extract effective seed points for dense correspondence. Correspondences are established by minimizing the bending energy between patches around seed points of given faces to those of a reference face. Given its hierarchical structure, our algorithm is capable of establishing thousands of correspondences between a large number of faces. Finally, a morphable model based on the dense corresponding points is fitted to an unseen query face for transfer of correspondences and hence automatic detection of landmarks. The proposed algorithm can detect any number of pre-defined landmarks including subtle landmarks that are even difficult to detect manually. Extensive experimental comparison on two benchmark databases containing 6, 507 scans shows that our algorithm outperforms six state of the art algorithms.

Results of the ICDAR 2015 Robust Reading Competition are presented. A new Challenge 4 on Incidental Scene Text has been added to the Challenges on Born-Digital Images, Focused Scene Images and Video Text. Challenge 4 is run on a newly acquired dataset of 1,670 images evaluating Text Localisation, Word Recognition and End-to-End pipelines. In addition, the dataset for Challenge 3 on Video Text has been substantially updated with more video sequences and more accurate ground truth data. Finally, tasks assessing End-to-End system performance have been introduced to all Challenges. The competition took place in the first quarter of 2015, and received a total of 44 submissions. Only the tasks newly introduced in 2015 are reported on. The datasets, the ground truth specification and the evaluation protocols are presented together with the results and a brief summary of the participating methods.

Hyperspectral imaging is emerging as a promising technology to discover patterns that are otherwise hard to identify with regular cameras. Recent research has shown the potential of hyperspectral image analysis to automatically distinguish visually similar inks. However, a major limitation of prior work is that automatic distinction only works when the number of inks to be distinguished is known a priori and their relative proportions in the inspected image are roughly equal. This research work aims at addressing these two problems. We show how anomaly detection combined with unsupervised clustering can be used to handle cases where the proportions of pixels belonging to the two inks are highly unbalanced. We have performed experiments on the publicly available UWA Hyperspectral Documents dataset. Our results show that INFLO anomaly detection algorithm is able to best distinguish inks for highly unbalanced ink proportions.

In this paper, we present a novel methodology for multiple script identification using Long Short-Term Memory (LSTM) networks' sequence-learning capabilities. Our method is able to identify multiple scripts at text-line level, where two or more scripts are present in the same text-line. Unlike traditional techniques, where either shape features or bounding boxes of individual characters are extracted, the LSTM-based system learns a particular script in a supervised learning framework. Moreover, this system neither needs specific features nor other preprocessing steps other than text-line extraction and text-line normalization. The proposed method works on text-line level, where it identifies each character as belonging to a particular script. We have developed a database consisting of English and Greek script, and our system achieved a script recognition accuracy of 98.186% on this dataset.

This paper introduces a novel curriculum learning strategy for ligature-based scripts. Long Short-Term Memory Networks require thousands or even millions of iterations on target symbols, depending upon the complexity of the target data, to converge when trained for sequence transcription because they have to localize the individual symbols along with the recognition. Curriculum learning reduces the number of target symbols to be visited before the network converges. In this paper, we propose a ligature-based complexity measure to define the sampling order of the training data. Experiments performed on UPTI database show that the curriculum learning using our strategy can reduce the total number of target symbols before convergence for printed Urdu Nastaleeq OCR task.

Spectra measured at a single pixel of a remotely sensed hyperspectral image is usually a mixture of multiple spectral signatures (endmembers) corresponding to different materials on the ground. Sparse unmixing assumes that a mixed pixel is a sparse linear combination of different spectra already available in a spectral library. It uses sparse approximation (SA) techniques to solve the hyperspectral unmixing problem. Among these techniques, greedy algorithms suite well to sparse unmixing. However, their accuracy is immensely compromised by the high correlation of the spectra of different materials. This paper proposes a novel greedy algorithm, called OMP-Star, that shows robustness against the high correlation of spectral signatures. We preprocess the signals with spectral derivatives before they are used by the algorithm. To approximate the mixed pixel spectra, the algorithm employs a futuristic greedy approach that, if necessary, considers its future iterations before identifying an endmember. We also extend OMP-Star to exploit the nonnegativity of spectral mixing. Experiments on simulated and real hyperspectral data show that the proposed algorithms outperform the state-of-the-art greedy algorithms. Moreover, the proposed approach achieves results comparable to convex relaxation-based SA techniques, while maintaining the advantages of greedy approaches.

A sparse principal component analysis (PCA) seeks a sparse linear combination of input features (variables), so that the derived features still explain most of the variations in the data. A group sparse PCA introduces structural constraints on the features in seeking such a linear combination. Collectively, the derived principal components may still require measuring all the input features. We present a joint group sparse PCA (JGSPCA) algorithm, which forces the basic coefficients corresponding to a group of features to be jointly sparse. Joint sparsity ensures that the complete basis involves only a sparse set of input features, whereas the group sparsity ensures that the structural integrity of the features is maximally preserved. We evaluate the JGSPCA algorithm on the problems of compressed hyperspectral imaging and face recognition. Compressed sensing results show that the proposed method consistently outperforms sparse PCA and group sparse PCA in reconstructing the hyperspectral scenes of natural and man-made objects. The efficacy of the proposed compressed sensing method is further demonstrated in band selection for face recognition.

Nowadays, Document forgery detection is becoming increasingly important as forgery techniques are becoming available even to untrained users. Hence, documents that do not contain any extrinsic security features (e.g. invoices) have become easier to forge. We previously presented a method to detect manipulated documents based on distortions introduced during the forgery creation process. In this paper, several approaches are explored to improve accuracy and time taken to detect forgeries based on document distortions. The main idea behind the presented approaches is to automatically identify which parts of a document belong to the template (and hence would remain static across different documents originating from the same source) and then detect distortions in those parts only. An improvement up to 29% in accuracy of forgery detection is observed compared to our previous work. Furthermore, we also present an approximation of the original method that results in a reduction in run time of the method by several orders of magnitude, while having only a marginal reduction in its accuracy.

Identifying the source printer of a document is important in forgery detection. The larger the number of documents to be investigated for forgery, the less time-efficient manual examination becomes. Assuming the document in question was scanned, the accuracy of automatic forgery detection depends on the scanning resolution. Low (100-200 dpi) and common (300-400 dpi) resolution scans have less distinctive features than high-end scanner resolution, whereas the former is more widespread in offices. In this paper, we propose a method to automatically identify source printers using common-resolution scans (400 dpi). Our method depends on distinctive noise produced by printers. Independent of the document content or size, each printer produces noise depending on its printing technique, brand and slight differences due to manufacturing imperfections. Experiments were carried out on a set of 400 documents of similar structure printed using 20 different printers. The documents were scanned at 400 dpi using the same scanner. Assuming constant settings of the printer, the overall accuracy of the classification was 76.75%.

Accurate knowledge of spectral reflectance is crucial for hyperspectral image analysis. We propose a novel spectral reflectance recovery method by adaptive spatio-spectral support. The proposed technique is evaluated in both simulated and real illumination scenarios. A multi-illuminant hyper-spectral scene database has been collected and made publicly available. Experiments show that the adaptive illuminant estimation reduces the mean angular error of recovered spectra by 13%.

Spectra measured at a pixel of a remote sensing hyper spectral sensor is usually a mixture of multiple spectra (end-members) of different materials on the ground. Hyper spectral unmixing aims at identifying the end members and their proportions (fractional abundances) in the mixed pixels. Hyper spectral unmixing has recently been casted into a sparse approximation problem and greedy sparse approximation approaches are considered desirable for solving it. However, the high correlation among the spectra of different materials seriously affects the accuracy of the greedy algorithms. We propose a greedy sparse approximation algorithm, called SUnGP, for unmixing of hyper spectral data. SUnGP shows high robustness against the correlation of the spectra of materials. The algorithm employees a subspace pruning strategy for the identification of the end members. Experiments show that the proposed algorithm not only outperforms the state of the art greedy algorithms, its accuracy is comparable to the algorithms based on the convex relaxation of the problem, but with a considerable computational advantage.

Hyperspectral images obtained from remote sensing platforms have limited spatial resolution. Thus, each spectra measured at a pixel is usually a mixture of many pure spectral signatures (endmembers) corresponding to different materials on the ground. Hyperspectral unmixing aims at separating these mixed spectra into its constituent end-members. We formulate hyperspectral unmixing as a constrained sparse coding (CSC) problem where unmixing is performed with the help of a library of pure spectral signatures under positivity and summation constraints. We propose two different methods that perform CSC repeatedly over the hyperspectral data. However, the first method, Repeated-CSC (RCSC), systematically neglects a few spectral bands of the data each time it performs the sparse coding. Whereas the second method, Repeated Spectral Derivative (RSD), takes the spectral derivative of the data before the sparse coding stage. The spectral derivative is taken such that it is not operated on a few selected bands. Experiments on simulated and real hyperspectral data and comparison with existing state of the art show that the proposed methods achieve significantly higher accuracy. Our results demonstrate the overall robustness of RCSC to noise and better performance of RSD at high signal to noise ratio.

Selecting a reduced set of relevant and non-redundant features for supervised classification problems is a challenging task. We propose a gradient based feature selection method which can search the feature space efficiently and select a reduced set of representative features. We test our proposed algorithm on five small and medium sized pattern classification datasets as well as two large 3D face datasets for computer vision applications. Comparison with the state of the art wrapper and filter methods shows that our proposed technique yields better classification results in lesser number of evaluations of the target classifier. The feature subset selected by our algorithm is representative of the classes in the data and has the least variation in classification accuracy.

Traditionally, stamps are considered as a seal of authenticity for documents. For automatic processing and verification, segmentation of stamps from documents is pivotal. Existing methods for stamp extraction mostly employ color and/or shape based techniques, thereby limiting their applicability to only colored and specific shape stamps. In this paper, a novel, generic method based on part-based features is presented for segmentation of stamps from document images. The proposed method can segment black, colored, unseen, arbitrary shaped, textual, as well as graphical stamps. The proposed method is evaluated on a publicly available dataset for stamp detection and verification and achieved recall and precision of 73% and 83% respectively, for black stamps which were not addressed in the past.

Long Short-Term Memory (LSTM) networks have yielded excellent results on handwriting recognition. This paper describes an application of bidirectional LSTM networks to the problem of machine-printed Latin and Fraktur recognition. Latin and Fraktur recognition differs significantly from handwriting recognition in both the statistical properties of the data, as well as in the required, much higher levels of accuracy. Applications of LSTM networks to handwriting recognition use two-dimensional recurrent networks, since the exact position and baseline of handwritten characters is variable. In contrast, for printed OCR, we used a one-dimensional recurrent network combined with a novel algorithm for baseline and x-height normalization. A number of databases were used for training and testing, including the UW3 database, artificially generated and degraded Fraktur text and scanned pages from a book digitization project. The LSTM architecture achieved 0.6% character-level test-set error on English text. When the artificially degraded Fraktur data set is divided into training and test sets, the system achieves an error rate of 1.64%. On specific books printed in Fraktur (not part of the training set), the system achieves error rates of 0.15% (Fontane) and 1.47% (Ersch-Gruber). These recognition accuracies were found without using any language modelling or any other post-processing techniques.

Text-line extraction is the backbone of document image analysis. Since decades, a large number of text-line finding methods have been proposed, where these methods rely on certain assumptions about a target class of documents with respect to writing styles, digitization methods, intensity values, and scripts. There is no generic text-line finding method that can be robustly applied to a large variety of simple and complex document images. We introduced the ridge-based text-line finding method, and published its initial results for curled text-line detection on camera-captured document images. In this paper, we demonstrates our ridge-based method as a generic text-line finding approach that can be robustly applied on a diverse collection of simple and complex document images. The comprehensive performance evaluation of the ridge-based method and its comparison with several state-of-the-art methods is presented in the paper. For this purpose, diverse categories of publicly available and standard datasets have been selected: UWIII (scanned, printed English script), DFKI-I (camera-captured, printed English script), UMD (handwritten Chinese, Hindi, and Korean scripts), ICDAR2007 handwritten segmentation contest (handwritten English, French, German and Greek scripts), Arabic/Urdu (scanned, printed script), and Fraktur (scanned, calligraphic German script). Experiments on these datasets show that the ridge-based method achieves better text-line extraction results as those of the best performing, domain-specific text-line finding methods. Firstly, these results show that the ridge-based method is a generic text-line extraction method. Secondly, these results are also helpful for the community to assess the advantages of this method.

This paper addresses the problem of holistic recognition of printed ligatures in Nastalique writing style of the Urdu language. The main difficulty of the recognition process lies in the large number of classes/ligatures (17,000 different possible ligatures in our Urdu text data). This large number of classes not only limits the efficiency (run-time) of the recognition algorithms, but also makes it difficult to use state-of-the-art classifiers - like Random Forests - that can only handle up to a few hundred classes. Nearest neighbor classifiers scale up well to tackle such large-scale classification problems, however their poor run-time efficiency poses a major obstacle. In this paper, we investigate two strategies for improving the efficiency (reducing the search space) of nearest neighbor based classification of Urdu ligatures. The first approach uses spectral hashing to resort to approximate nearest neighbor classification. The second approach is based on the idea of hierarchical classification to partition the search space based on the number of characters in a ligature. Experiments using spectral hashing show that the search space of nearest neighbor comparison can be reduced by about 50% without a loss in recognition accuracy. Further experiments demonstrate that the Random Forest classifier can be reliably used as the first stage classifier to distinguish one-character ligatures from multiple-character ligatures in a hierarchical classification scheme. We hope that the ideas presented in this paper would build the foundations for practical large-scale ligature classification systems not only for Nastalique, but also for other Urdu and Arabic scripts.

Automatically identifying that a certain page in a set of documents is printed with a different printer than the rest of the documents can give an important clue for a possible forgery attempt. Different printers vary in their produced printing quality, which is especially noticeable at the edges of printed characters. In this paper, a system using the difference in edge roughness to distinguish laser printed ages from inkjet printed pages is presented. Several feature extraction methods have been developed and evaluated for that purpose. In contrast to previous work, this system uses unsupervised anomaly detection to detect documents printed by a different printing technique than the majority of the documents among a set. This approach has the advantage that no prior training using genuine documents has to be done. Furthermore, we created a dataset featuring 1200 document images from different domains (invoices, contracts, scientific papers) printed by 7 different inkjet and 13 laser printers. Results show that the presented feature extraction method achieves the best outlier rank score in comparison to state-of-the-art features.

This report presents the final results of the ICDAR 2013 Robust Reading Competition. The competition is structured in three Challenges addressing text extraction in different application domains, namely born-digital images, real scene images and real-scene videos. The Challenges are organised around specific tasks covering text localisation, text segmentation and word recognition. The competition took place in the first quarter of 2013, and received a total of 42 submissions over the different tasks offered. This report describes the datasets and ground truth specification, details the performance evaluation protocols used and presents the final results along with a brief summary of the participating methods.

Handwritten signatures are one of the most socially acceptable and traditionally used person identification and authentication metric. Although a number of authentication systems based on handwritten signatures have been proposed, a little attention is paid towards employing signatures for person identification. In this work, we address both the identification and verification problems related to analysis of dynamic handwritten signatures. In this way, the need to present username before biometric verification can be eliminated in current signature based biometric authentication systems. A compressed sensing approach is used for user identification and to reject a query signature that does not belong to any user in the database. Once a person is identified, an automatic alignment of query signature with the reference template is carried out such that the correlations between two signature instances are maximized. An elastic distance matching algorithm is then run over the presented data which declares the query signature as either genuine or forged based on the dissimilarity with the reference signature. Our results show that dynamic signatures can be accurately used for person identification along with the traditional verification methods.

Ink mismatch detection provides important clues to forensic document examiners by identifying whether a particular handwritten note was written with a specific pen, or to show that some part (e.g. signature) of a note is written with a different ink as compared to the rest of the note. In this paper, we show that a hyper spectral image (HSI) of handwritten notes can discriminate between inks that are visually similar in appearance. For this purpose, we develop the first ever hyper spectral image database of handwritten notes in various blue and black inks, comprising a total of 70 hyper spectral images each in 33 bands of the visible spectrum. In an unsupervised clustering scheme, the spectral responses of inks fall into separate clusters to allow segmentation of two different inks in a questioned document. The same method fails to segment inks correctly when applied to RGB scans of these documents, since the inks are very hard to distinguish in the visible spectral range. HSI overcomes the shortcomings of RGB and allows better discrimination between inks. We further evaluate which subset of bands from HSI is most useful for the purpose of ink mismatch detection. We hope that these findings will stimulate the use of HSI in document analysis research, especially for questioned document examination.

Recurrent neural networks (RNN) have been successfully applied for recognition of cursive handwritten documents, both in English and Arabic scripts. Ability of RNNs to model context in sequence data like speech and text makes them a suitable candidate to develop OCR systems for printed Nabataean scripts (including Nastaleeq for which no OCR system is available to date). In this work, we have presented the results of applying RNN to printed Urdu text in Nastaleeq script. Bidirectional Long Short Term Memory (BLSTM) architecture with Connectionist Temporal Classification (CTC) output layer was employed to recognize printed Urdu text. We evaluated BLSTM networks for two cases: one ignoring the character's shape variations and the second is considering them. The recognition error rate at character level for first case is 5.15% and for the second is 13.6%. These results were obtained on synthetically generated UPTI dataset containing artificially degraded images to reflect some real-world scanning artifacts along with clean images. Comparison with shape-matching based method is also presented.

Given the growing amount of very large image databases, content based image retrieval (CBIR) is becoming more and more important. One of the major challenges in CBIR is the semantic gap — commonly used feature-based algorithms are not able to identify what really draws human attention in an image. This problem is more crucial for localized CBIR, where certain regions / parts of the image are what the user is really interested in. This paper explores how human gaze can be utilized to extract regions of interest (ROIs) of an image to perform attention based image retrieval. Using eye tracking data and knowledge about foveal and peripheral vision of humans, we present a foveal fixation clustering algorithm that automatically generates ROIs in an image while a person is viewing it. To objectively set different parameters of the algorithm, a small user study was conducted. The method was evaluated for use in a localized CBIR system. Image retrieval results using the publicly available SIVAL dataset were scored using mean average precision (MAP). Comparison to a saliency-based visual attention algorithm as well as to manually labeled regions showed that the retrieval results of the developed algorithm are nearly two times better than the saliency-based visual attention algorithm and very close to the results using hand-labeled regions.

Document images prove to be a difficult case for standard stereo correspondence approaches. One of the major problem is that document images are highly self-similar. Most algorithms try to tackle this problem by incorporating a global optimization scheme, which tends to be computationally expensive. In this paper, we show that incorporation of layout information into the matching paradigm, as a grouping entity for features, leads to better results in terms of robustness, efficiency, and ultimately in a better 3D model of the captured document, that can be used in various document restoration systems. This can be seen as a divide and conquer approach that partitions the search space into portions given by each grouping entity and then solves each of them independently. As a grouping entity text-lines are preferred over individual character blobs because it is easier to establish correspondences. Text-line extraction works reasonably well on stereo image pairs in the presence of perspective distortions. The proposed approach is highly efficient and matches obtained are more reliable. The claims are backed up by showing their practical applicability through experimental evaluations.

Recognition of scene text using a hand-held camera is emerging as a hot topic of research. In this paper, we investigate the use of a head-mounted eye-tracker for scene text recognition. An eye-tracker detects the position of the user's gaze. Using gaze information of the user, we can provide the user with more information about his region/object of interest in a ubiquitous manner. Therefore, we can realize a service such as the user gazes at a certain word and soon obtain the related information of the word by combining a word recognition system with eye-tracking technology. Such a service is useful since the user has to do nothing but gazes at interested words. With a view to realize the service, we experimentally evaluate the effectiveness of using the eye-tracker for word recognition. The initial results show the recognition accuracy was around 70% in our word recognition experiment and the average computational time was less than one second per a query image.

Optical character recognition (OCR) of machine printed Latin script documents is ubiquitously claimed as a solved problem. However, error free OCR of degraded or noisy text is still challenging for modern OCR systems. Most recent approaches perform segmentation based character recognition. This is tricky because segmentation of degraded text is itself problematic. This paper describes a segmentation free text line recognition approach using multi layer perceptron (MLP) and hidden markov models (HMMs). A line scanning neural network-trained with character level contextual information and a special garbage class-is used to extract class probabilities at every pixel succession. The output of this scanning neural network is decoded by HMMs to provide character level recognition. In evaluations on a subset of UNLV-ISRI document collection, we achieve 98.4% character recognition accuracy that is statistically significantly better in comparison with character recognition accuracies obtained from state-of-the-art open source OCR systems.

Computational models of visual attention use image features to identify salient locations in an image that are likely to attract human attention. Attention models have been quite effectively used for various object detection tasks. However, their use for scene text detection is under-investigated. As a general observation, scene text often conveys important information and is usually prominent or salient in the scene itself. In this paper, we evaluate four state-of-the-art attention models for their response to scene text. Initial results indicate that saliency maps produced by these attention models can be used for aiding scene text detection algorithms by suppressing non-text regions.

Table of Contents (ToC) is an integral part of multiple-page documents like books, magazines, etc. Most of the existing techniques use textual similarity for automatically detecting ToC pages. However, such techniques may not be applied for detection of ToC pages in situations where OCR technology is not available, which is indeed true for historical documents and many modern Nabataean (Arabic) and Indic scripts. It is, therefore, necessary to develop tools to navigate through such documents without the use of OCR. This paper reports a preliminary effort to address this challenge. The proposed algorithm has been applied to find Table of Contents (ToC) pages in Urdu books and an overall initial accuracy of 88% has been achieved.

Wearable eye trackers open up a large number of opportunities to cater for the information needs of users in today's dynamic society. Users no longer have to sit in front of a traditional desk-mounted eye tracker to benefit from the direct feedback given by the eye tracker about users' interest. Instead, eye tracking can be used as a ubiquitous interface in a real-world environment to provide users with supporting information that they need. This paper presents a novel application of intelligent interaction with the environment by combining eye tracking technology with real-time object recognition. In this context we present i) algorithms for guiding object recognition by using fixation points ii) algorithms for generating evidence of users' gaze on particular objects iii) building a next generation museum guide called Museum Guide 2.0 as a prototype application of gaze-based information provision in a real-world environment. We performed several experiments to evaluate our gaze-based object recognition methods. Furthermore, we conducted a user study in the context of Museum Guide 2.0 to evaluate the usability of the new gaze-based interface for information provision. These results show that an enormous amount of potential exists for using a wearable eye tracker as a human-environment interface.

Document forgery detection is important as techniques to generate forgeries are becoming widely available and easy to use even for untrained persons. In this work, two types of forgeries are considered: forgeries generated by re-engineering a document and forgeries that are generated using scanning and printing a genuine document. An unsupervised approach is presented to automatically detect forged documents of these types by detecting the geometric distortions introduced during the forgery process. Using the matching quality between all pairs of documents, outlier detection is performed on the summed matching quality to identify the tampered document. Quantitative evaluation is done on two public data sets, reporting a true positive rate from to 0.7 to 1.0.

Text-line extraction is a key task in document analysis. Methods based on an isotropic Gaussian filtering and ridge detection have shown good results. This paper describes performance improvements to these technique based on the use of a convolution of isotropic Gaussian filter with line filters. These new filter banks are motivated by a matched filter approach to text-lines and, in addition, require fewer operations to compute. We evaluate the performance of the new filter bank in combination with ridge detection on the public DFKI-I (CBDAR 2007 dewarping contest) dataset, which contains camera captured document images and demonstrate improvements in performance to previous state-of-the-art techniques.

Text-lines extraction and their reading order determination is an important step in optical character recognition (OCR) systems. Research in OCR of Arabic script documents has primarily focused on character recognition and therefore most of researchers use primitive methods like projection profile analysis for text-line extraction. Although projection methods achieve good accuracy on clean, skew corrected documents, their performance drops under challenging situations (border noise, skew, complex layouts). This paper presents a robust layout analysis system for extracting text-lines in reading order from scanned Arabic script document images written in different languages (Arabic, Urdu, Persian) and styles (Naskh, Nastaliq). The presented system is based on a suitable combination of different well established techniques for analyzing Latin script documents that have proven to be robust against different types of document image degradations. Evaluation of the presented system on Arabic and Urdu document image datasets consisting of a variety of complex single- and multi-column layouts achieves high accuracies for text and non-text segmentation, text-line extraction, and reading order determination.

Segmentation and recognition of screen rendered text is a challenging task due to its low resolution (72 or 96 ppi) and use of antialiased rendering. This paper evaluates Hidden Markov Model (HMM) techniques for OCR of low resolution text -- both on screen rendered isolated characters and screen rendered text-lines -- and compares it with the performance of other commercial and open source OCR systems. Results show that HMM-based methods reach the performance of other methods on screen rendered text and yield above 98% character level accuracies on both screen rendered text-lines and characters.

Time-stamps and URLs overlaid artificially on images add useful meta information which can be used for automatic indexing of images and videos. In this paper, we propose a method based on an attention-based model of visual saliency to extract overlaid text and time-stamps that are rendered on images. Our model of visual saliency is based on a Bayesian framework and works very well for the task of time-stamp detection and segmentation as is evident by overall object recall of 80% and precision of 70%. Our method produces a clean text segmented binarized image, which can be used for recognition directly by an OCR system. Furthermore, our technique is robust against variation of font styles and color of time-stamp and overlaid text.

Recognition of text in natural scene images is becoming a prominent research area due to the widespread availablity of imaging devices in low-cost consumer products like mobile phones. To evaluate the performance of recent algorithms in detecting and recognizing text from complex images, the ICDAR 2011 Robust Reading Competition was organized. Challenge 2 of the competition dealt specifically with detecting/recognizing text in natural scene images. This paper presents an overview of the approaches that the participants used, the evaluation measure, and the dataset used in the Challenge 2 of the contest. We also report the performance of all participating methods for text localization and word recognition tasks and compare their results using standard methods of area precision/recall and edit distance.

Projection methods have been used in the analysis of bitonal document images for different tasks such as page segmentation and skew correction for more than two decades. However, these algorithms are sensitive to the presence of border noise in document images. Border noise can appear along the page border due to scanning or photocopying. Over the years, several page segmentation algorithms have been proposed in the literature. Some of these algorithms have come into widespread use due to their high accuracy and robustness with respect to border noise. This paper addresses two important questions in this context: 1) Can existing border noise removal algorithms clean up document images to a degree required by projection methods to achieve competitive performance? 2) Can projection methods reach the performance of other state-of-the-art page segmentation algorithms (e.g., Docstrum or Voronoi) for documents where border noise has successfully been removed? We perform extensive experiments on the University of Washington (UW-III) data set with six border noise removal methods. Our results show that although projection methods can achieve the accuracy of other state-of-the-art algorithms on the cleaned document images, existing border noise removal techniques cannot clean up documents captured under a variety of scanning conditions to the degree required to achieve that accuracy.

Passports, ID cards, banknotes, and degrees are considered as valuable documents that need to be secured against forgery. Apart from those, there are many other document types that are valuable, too, but that do not have any security features, as e.g. bills and vouchers. These may be used by fraudulent people to defraud money from e.g. a car insurance company. The wide availability of scanning and printing hardware allows even non-experts to easily forge a document. We therefore present a new aspect in the examination of intrinsic document features for optical document security: the goal is to automatically detect text-lines that have been manipulated or additionally inserted in a document by inspecting their alignment (left, right or center) with respect to the other text-lines in the document. This constitutes an additional feature in the goal of developing a powerful toolbox for automatic document inspection. Using the extracted text-lines, the alignment margins are extracted. Statistics on the distances of the text-lines to the alignment margins are used to identify lines that might have been forged. Such documents can then be presented to a human operator for further inspection. Due to lack of public datasets containing forged documents, a new dataset had to be created. Evaluation showed a classification accuracy of 90.5%.

Segmentation of a document image into text and non-text regions is an important preprocessing step for a variety of document image analysis tasks, like improving OCR, document compression etc. Most of the state-of-the-art document image segmentation approaches perform segmentation using pixel-based or zone(block)-based classification. Pixel-based classification approaches are time consuming, whereas block-based methods heavily depend on the accuracy of block segmentation step. In contrast to the state-of-the-art document image segmentation approaches, our segmentation approach introduces connected component based classification, thereby not requiring a block segmentation beforehand. Here we train a self-tunable multi-layer perceptron (MLP) classifier for distinguishing between text and non-text connected components using shape and context information as a feature vector. Experimental results prove the effectiveness of our proposed algorithm. We have evaluated our method on subset of UW-III, ICDAR 2009 page segmentation competition test images and circuit diagrams datasets and compared its results with the state-of-the-art leptonica's page segmentation algorithm.

Detecting tables in document images is important since not only do tables contain important information, but also most of the layout analysis methods fail in the presence of tables in the document image. Existing approaches for table detection mainly focus on detecting tables in single columns of text and do not work reliably on documents with varying layouts. This paper presents a practical algorithm for table detection that works with a high accuracy on documents with varying layouts (company reports, newspaper articles, magazine pages, ...). An open source implementation of the algorithm is provided as part of the Tesseract OCR engine. Evaluation of the algorithm on document images from publicly available UNLV dataset shows competitive performance in comparison to the table detection module of a commercial OCR system.

Table spotting and structural analysis are just a small fraction of tasks relevant when speaking of table analysis. Today, quite a large number of different approaches facing these tasks have been described in literature or are available as part of commercial OCR systems that claim to deal with tables on the scanned documents and to treat them accordingly.

High quality conversions of scanned documents into PDF usually either rely on full OCR or token compression. This paper describes an approach intermediate between those two: it is based on token clustering, but additionally groups tokens into candidate fonts. Our approach has the potential of yielding OCR-like PDFs when the inputs are high quality and degrading to token based compression when the font analysis fails, while preserving full visual fidelity. Our approach is based on an unsupervised algorithm for grouping tokens into candidate fonts. The algorithm constructs a graph based on token proximity and derives token groups by partitioning this graph. In initial experiments on scanned 300 dpi pages containing multiple fonts, this technique reconstructs candidate fonts with 100% accuracy.

Camera-captured document images often contain curled textlines because of geometric and perspective distortions. Finding curled textlines, which is more difficult than straight textline detection, is a primary step in the processing of hand-held camera-captured document images. Detected textlines results can be used for dewarping of warped images, layout-analysis, etc. In this paper, we compare previously reported curled textline segmentation techniques by using the publicly available CBDAR 2007 dewarping contest dataset and vectorial performance evaluation metrics.

Document script recognition is one of the important preprocessing steps in a multilingual optical character recognition (MOCR) system. A MOCR system requires prior knowledge of script to accurately recognize multilingual text in a single document. In multilingual documents two scripts can be mixed together within a single text line. Many existing script recognition methods lack the ability to recognize multiple scripts mixed within a single text line. Besides, these methods usually use script dependent features for script recognition thereby limiting their scope to particularly that script. In this paper we propose a discriminative learning approach for multi-script recognition at connected component level by using a convolutional neural network. The convolutional neural network combines feature extraction and script recognition process in one step and discriminative features for script recognition are extracted and learned as convolutional kernels from raw input. This eliminates the need for manually defining discriminative features for particular scripts. Results show above 95% script recognition accuracy at connected component level on datasets of Greek-Latin, Arabic-Latin multi-script documents and Antiqua-Fraktur documents. The proposed method can be easily adapted to different scripts.

Detection of curled textline is important for dewarping of hand-held camera-captured document images. Then baselines and the lines following the top of x-height of characters (x-lines) are estimated for dewarping. Existing curled textline segmentation approaches are sensitive to outlier points and perspective distortions. Furthermore these approaches use regression over top and bottom points of a segmented textline to estimate its x-line and baseline separately, which may results in inaccurate estimation. Here we propose a novel curled textline segmentation approach based on active contours (snakes) in which we perform segmentation by estimating the pairs of x-line and baseline; solving both problems together. Starting form a connected component we jointly trace a pair of x-line and baseline using coupled snakes and external energies of neighboring top-bottom points. We grow neighborhood region iteratively during tracing, which results in robustness to perspective distortions, and maintain a natural property of similar distance within the pair of x-line and baseline pair, which results in robustness to outlier points. We achieved 90.76% of one-to-one match-score recognition accuracy of curled textline segmentation on CBDAR 2007 document image dewarping contest dataset, with good estimation of pairs of x-line and baseline.

Handwritten document images contain textlines with multi orientations, touching and overlapping characters within consecutive textlines, and small inter-line spacing making textline segmentation a difficult task. In this paper we propose a novel, script-independent textline segmentation approach for handwritten documents, which is robust against above mentioned problems. We model textline extraction as a general image segmentation task. We compute the central line of parts of textlines using ridges over the smoothed image. Then we adapt the state-of-the-art active contours (snakes) over ridges, which results in textline segmentation. Unlike the "level set'' and "Mumford-Shah model'' based handwritten textline segmentation methods, our method use matched filter bank approach for smoothing and does not require heuristic post processing steps for merging or splitting segmented textlines. Experimental results prove the effectiveness of the proposed algorithm. We evaluated our algorithm on ICDAR 2007 handwritten segmentation contest dataset and obtained an accuracy of 96.3%.

Cameras offer flexible document imaging, but with uneven shading and non-planar page shape. Therefore camera captured documents need to go through dewarping before being processed by traditional text recognition methods. Curled textline detection is an important step of dewarping. Previous approaches of curled textline detection use binarization as a pre-processing step, which can negatively affect the detection results under uneven shading. Furthermore, these approaches are sensitive to high degrees of curl and estimate x-line1 and baseline pairs using regression which may result in inaccurate estimation. We introduce a novel curled textline detection approach for grayscale document images. First, the textline structure is enhanced by using match filter bank smoothing and then central lines of textlines are detected using ridges. Then, x-line and baseline pairs are estimated by adapting active contours (snakes) over ridges. Unlike other approaches, our approach does not use binarization and applies directly on grayscale images. We achieved 91% of detection accuracy with good estimation of x-line and baseline pairs on the dataset of CBDAR 2007 document image dewarping contest.

In document image recognition, orientation detection of the scanned page is necessary for the following procedures to work correctly as they assume that the text is well oriented. Several methods have been proposed, but most of them rely on heuristics of the script such as the graphical asymmetry between ascenders and descenders for Roman script. The literature shows that as soon as this assumption is not fulfilled, e.g. plain capital text, noisy or degraded characters, etc. they fail. For a large-scale digitalization process, a low error and rejection rate are expected in order to reduce the amount of human intervention. We propose a Recognition Driven Page Orientation Detection (RD-POD) which does not depend on external criteria or assumption on the shape of the script. It uses the OCR engine for estimating the right orientation with a few lines of the document image. The RD-POD is highly robust and accurate, and is able to detect multiple orientations. Experimental evaluation shows that our method outperforms the current state-of-the-art on UW-1 dataset with an accuracy of 99.7%. Further tests on other three large and public datasets (MARG, ICDAR07, Google 1000 books) show accuracies of above 99% on each of them.

In contrast to prior experimental work, our results support the conclusion that RXYC can perform well after marginal noise removal. However, marginal noise removal on page images like those found in UW3 remains a hard problem, and it therefore remains an open question whether RXYC can actually achieve competitive performance on such databases.

Most optical character recognition (OCR) systems need to be trained and tested on the symbols that are to be recognized. Therefore, ground truth data is needed. This data consists of character images together with their ASCII code. Among the approaches for generating ground truth of real world data, one promising technique is to use electronic version of the scanned documents. Using an alignment method, the character bounding boxes extracted from the electronic document are matched to the scanned image. Current alignment methods are not robust to different similarity transforms. They also need calibration to deal with non-linear local distortions introduced by the printing/scanning process. In this paper we present a significant improvement over existing methods, allowing to skip the calibration step and having a more accurate alignment, under all similarity transforms. Our method finds a robust and pixel accurate scanner independent alignment of the scanned image with the electronic document, allowing the extraction of accurate ground truth character information. The accuracy of the alignment is demonstrated using documents from the UW3 dataset. The results show that the mean distance between the estimated and the ground truth character bounding box position is less than one pixel.

Segmentation of curled textlines from warped document images is one of the major issues in document image dewarping. Most of the curled textlines segmentation algorithms present in the literature today are sensitive to the degree of curl, direction of curl, and spacing between adjacent lines. We present a new algorithm for curled textline segmentation which is robust to above mentioned problems at the expense of high execution time. We will demonstrate this insensitivity in a performance evaluation section. Our approach is based on the state-of-the-art image segmentation technique: Active Contour Model (Snake) with the novel idea of several baby snakes and their convergence in a vertical direction only. Experiment on publically available CBDAR 2007 document image dewarping contest dataset shows our text line segmentation algorithm accuracy of 97.96%.

A key limitation of current layout analysis methods is that they rely on many hard-coded assumptions about document layouts and can not adapt to new layouts for which the underlying assumptions are not satisfied. Another major drawback of these approaches is that they do not return confidence scores for their outputs. These problems pose major challenges in large scale digitization efforts where a large number of different layouts need to be handled and manual inspection of the results on each individual page is not feasible. This paper presents a novel statistical approach to layout analysis that aims at solving the above mentioned problems for Manhattan layouts. The presented approach models known page layouts as a structural mixture model. A probabilistic matching algorithm is presented that gives multiple interpretations of input layout with associated probabilities. First experiments on documents from the publicly available MARG dataset achieved below 5%error rate for geometric layout analysis.

Geometric layout analysis plays an important role in document image understanding. Many algorithms known in literature work well on standard document images, achieving high text line segmentation accuracy on the UW-III dataset. These algorithms rely on certain assumptions about document layouts, and fail when their underlying assumptions are not met. Also, they do not provide confidence scores for their output. These two problems limit the usefulness of general purpose layout analysis methods in large scale applications. In this contribution, we propose a statistically motivated model-based trainable layout analysis system that allows assumption-free adaptation to different layout types and produces likelihood estimates of the correctness of the computed page segmentation. The performance of our approach is tested on a subset of the Google 1000 books dataset where it achieved a text line segmentation accuracy of 98.4% on layouts where other general-purpose algorithms failed to do a correct segmentation.

Informative benchmarks are crucial for optimizing the page segmentation step of an OCR system, frequently the performance limiting step for overall OCR system performance. We show that current evaluation scores are insufficient for diagnosing specific errors in page segmentation and fail to identify some classes of serious segmentation errors altogether. This paper introduces a vectorial score that is sensitive to, and identifies, the most important classes of segmentation errors (over-, under-, and mis-segmentation) and what page components (lines, blocks, etc.) are affected. Unlike previous schemes, our evaluation method has a canonical representation of ground truth data and guarantees pixel-accurate evaluation results for arbitrary region shapes. We present the results of evaluating widely used segmentation algorithms (x-y cut, smearing, whitespace analysis, constrained text-line finding, docstrum, and Voronoi) on the UW-III database and demonstrate that the new evaluation scheme permits the identification of several specific flaws in individual segmentation methods.

This paper presents a flexible and effective example- based approach for labeling title pages which can be used for automated extraction of bibliographic data. The labels of interest are "title", "author", "abstract" and "affiliation". The method takes a set of labeled document layouts and a single unlabeled document layout as input and finds the best matching layout in the set. The labels of this layout are used to label the new layout. The similarity measure for layouts combines structural layout similarity and textural similarity on the block-level. Experimental results yield accuracy rates from 94.8% to 99.6% obtained on the publicly available MARG dataset. This shows that our lightweight method has equivalent and partially better performance when compared to other more complex labeling methods known from the literature.

In this paper we present an approach for lip motion analysis that can be used in conjunction with a person authentication system based on face recognition, to avoid attacks on the system using passive photographs. This work focuses on robustly tracking lips in gray scale images, which may be captured in the visible light or near infrared spectrum. We present an approach for locating the two lip corners in a face image. Then we extract suitable features from the mouth region to classify mouth states (visemes). The system shows a classification accuracy of above 85%. The temporal changes in the detected viseme classes can be used for detecting the imposter.

This paper presents a two-step vision-based approach for real time inspection of the assembled valve keys in automobile cylinder heads. In the first step a combination of Sobel edge detector with gradient pair vector method is used for the detection of circular shaped valve keys. This method is very fast but also highly sensitive to noise, resulting in a relatively high percentage of false negatives. The second inspection step employs Canny edge detector with circular Hough transform. The images of the valve keys rejected in the first step pass through the second inspection step. This procedure differentiates between the false negatives and the true negatives. The cylinder heads rejected in the second step are separated afterwards from the main stream of the transportation line. In the statistics obtained, the first inspection step successfully rejected all of the wrongly assembled valve keys. The false rejection rate was around 4.6% of the total number of the valve keys rejected. The second inspection step successfully differentiated between true negatives and false negatives, reducing false negatives to 0%.