In this paper we present AMPNet, an acoustic abnormality detection model deployed at ACV Auctions to automatically identify engine faults of vehicles listed on the ACV Auctions platform. We investigate the problem of engine fault detection and discuss our approach of deep-learning based audio classification on a large-scale automobile dataset collected at ACV Auctions. Specifically, we discuss our data collection pipeline and its challenges, dataset preprocessing and training procedures, and deployment of our trained models into a production setting. We perform empirical evaluations of AMPNet and demonstrate that our framework is able to successfully capture various engine anomalies agnostic of vehicle type. Finally we demonstrate the effectiveness and impact of AMPNet in the real world, specifically showing a 20.85% reduction in vehicle arbitrations on ACV Auctions' live auction platform.

This work summarizes the results of the first Competition on Harvesting Raw Tables from Infographics (ICDAR 2019 CHART-Infographics). The complex process of automatic chart recognition is divided into multiple tasks for the purpose of this competition, including Chart Image Classification (Task 1), Text Detection and Recognition (Task 2), Text Role Classification (Task 3), Axis Analysis (Task 4), Legend Analysis (Task 5), Plot Element Detection and Classification (Task 6.a), Data Extraction (Task 6.b), and End-to-End Data Extraction (Task 7). We provided a large synthetic training set and evaluated submitted systems using newly proposed metrics on both synthetic charts and manually-annotated real charts taken from scientific literature. A total of 8 groups registered for the competition out of which 5 submitted results for tasks 1-5. The results show that some tasks can be performed highly accurately on synthetic data, but all systems did not perform as well on real world charts. The data, annotation tools, and evaluation scripts have been publicly released for academic use.

Online lecture videos are increasingly important e-learning materials for students. Automated content extraction from lecture videos facilitates information retrieval applications that improve access to the lecture material. A significant number of lecture videos include the speaker in the image. Speakers perform various semantically meaningful actions during the process of teaching. Among all the movements of the speaker, key actions such as writing or erasing potentially indicate important features directly related to the lecture content. In this paper, we present a methodology for lecture video content extraction using the speaker actions. Each lecture video is divided into small temporal units called action segments. Using a pose estimator, body and hands skeleton data are extracted and used to compute motion-based features describing each action segment. Then, the dominant speaker action of each of these segments is classified using Random forests and the motion-based features. With the temporal and spatial range of these actions, we implement an alternative way to draw key-frames of handwritten content from the video. In addition, for our fixed camera videos, we also use the skeleton data to compute a mask of the speaker writing locations for the subtraction of the background noise from the binarized key-frames. Our method has been tested on a publicly available lecture video dataset, and it shows reasonable recall and precision results, with a very good compression ratio which is better than previous methods based on content analysis.

The goal of a writer adaptive handwriting recognition system is to build a model that improves the recognition of a generic recognition model for a specific author. In this work, we show how structural representation learned from a generic writer-independent handwriting recognition model can be customized to individual authors. Convolutional Neural Network has shown outstanding performance in learning image-based representation that was used for classification. Additionally, they have been used along with Recurrent Neural Network (RNN) or its variations like, LSTM and GRU layers to analyze and understand sequences in handwriting recognition, sentence analysis, voice recognition etc. In most cases, the CNNs serve as a feature extractor instead of low-level hand-designed features that were used previously for the above-mentioned classification tasks. We design a method to reuse weights from layers trained on the IAM offline handwritten dataset to compute mid-level image representation for text in the Washington and Moore dataset. We show that despite differences in the writing style, fonts across these datasets, the transferred representation is able to capture a spatio-temporal representation leading to significantly improved recognition results. We hypothesize that the performance is solely not dependent on the number of samples and the model is evaluated with varying amount of fine-tuning samples showing promising results backing the hypothesis.

This paper describes a model based framework for detection and extraction of the contents of table cells from degraded handwritten document images that contain tables. Given the very poor quality of the target documents, the table cell detection problem is formulated conceptually as a two-step process. The first step is to identify the location of the table and extract the content of table cells given a model of the structure of the table present in the image. The second step is to identify the model of the table present in a document image from a list of given table models. A model-based representation for tables is introduced and is used for matching table candidates with the given model to identify and extract the contents of table cells. The approach for detecting potential table candidates is based on the detection of horizontal and vertical table line candidates. The table representation is a matrix of horizontal and vertical table line crossings, and the matching algorithm is formulated as a minimization problem where the optimal table candidate is obtained using the minimal distance between the candidate and model table matrices which is then used for extraction of the table cell contents. A similar approach is used to solve the model selection problem where the best fitting location in the document page for each of the candidate models is identified using the distance minimization approach along with a confidence score and the model with the highest confidence score is selected as the correct model. The approach was tested on document page images containing tables from the challenge set of the DARPA MADCAT handwritten document image data. Results indicate that the method is effective for both model selection as well as table cell content extraction.

In this paper we present a new dual mode, twin-folio structured English handwriting dataset IBM_UB_1. IBM_UB_1 is our first major release from a large multilingual handwriting corpus. Containing over 6000 pages of handwritten matter, this dataset can not only be used for unconstrained handwriting recognition, more importantly, the dataset's unique twin-folio structure presents a natural fit for research on writer identification, keyword spotting, indexing and various forms of handwritten document search and retrieval. We first describe two central characteristics of the dataset - the twin-folio structure and dual modality (online/offline) - and their relevance to current research problems. Secondly, we describe the dataset, its collection and construction, and provide key descriptive statistics. Finally, we evaluate the dataset on two different research domains - handwriting recognition and writer identification - and present related experimental results.

We propose a segmentation based online word recognition approach which uses a Conditional Random Field (CRF) driven beam search strategy. An efficient trie-lexicon directed, breadth-first beam search algorithm is employed in a combined segmentation-and-recognition framework to accomplish real-time recognition of online handwritten cursive English words. This framework is developed by building a candidate lattice of primitive segments obtained through over segmentation of the word pattern. The search space for the lattice is expanded by synchronously matching the lattice nodes to likely character patterns from a trie-dictionary constructed out of the target lexicon. The probable paths are evaluated by integrating character recognition scores with physical and spatial characteristics of the handwritten segments in a CRF (conditional random field) model and a beam search strategy is used to prune the set of likely paths. This approach has been benchmarked on the new IBM_UB_1 dataset as well as on the UNIPEN dataset for comparison.

This paper describes an online handwritten English cursive word recognition method using a segmentation-free Markov random field (MRF) model in combination with an offline recognition method which uses pseudo 2D bi-moment normalization (P2DBMN) and modified quadratic discriminant function (MQDF). It extracts feature points along the pen-tip trace from pen-down to pen-up and uses the feature point coordinates as unary features and the differences in coordinates between the neighboring feature points as binary features. Each character is modeled as a MRF and word MRFs are constructed by concatenating character MRFs according to a trie lexicon of words during recognition. Our method expands the search space using a character-synchronous beam search strategy to search the segmentation and recognition paths. This method restricts the search paths from the trie lexicon of words and preceding paths, as well as the lengths of feature points during path search. We also combine it with a P2DBMN-MQDF recognizer that is widely used for Chinese and Japanese character recognition.

This paper presents a novel set of image enhancement algorithms for binary images of poorly scanned real world page documents. Problems that are targeted by the methods described include large blobs or clutter noise, salt-and-pepper noise and detection and removal of non-text objects such as form lines or rule-lines. The algorithms described are shown to be very effective in removing clutter noise and pepper noise as well as form lines and rule-lines. A region growing algorithm is also described to enhance the quality of the text and to fix the problems arising from the salt noise which leaves holes in the text and creates broken strokes. The methods were tested on 204 images from the challenge set of the DARPA MADCAT Arabic handwritten document image data. The results indicate that the methods described are robust and are capable of significantly improving the image quality for downstream OCR systems.

In this paper, we describe a novel approach to Writer Identification in Offline handwriting using Latent Dirichlet Allocation. State-of-the-art methods for writer identification employ the traditional feature-classification paradigm which does not provide enough information about the handwriting attributes such as writing style which are key components in any forensic analysis of handwriting. This problem is also compounded due to lack of efficient rules for defining a particular writing style that can capture writer specific characteristics over a large dataset. We propose to address this issue by using a generative model in form of Latent Dirichlet Allocation(LDA) that automatically infers writing styles from handwritten document collection without any pre-defined set of rules. This information is then used to represent each writer as a distribution over multiple writing style for classifying any unknown writer sample. We describe our approach on two different feature sets consisting of contour angle features as well as structural and concavity features. Our experimental results show comparable performance with baseline systems and also demonstrate the efficacy of LDA for learning multiple handwriting styles.

Separating machine printed text and handwriting from overlapping text is a challenging problem in the document analysis field and no reliable algorithms have been developed thus far. In this paper, we propose a novel approach for separating handwriting from binary image of overlapped text. Instead of using fixed size training patches, we describe an aggregation method which uses shape context features to extract training samples automatically. We use a Markov Random Field (MRF) to model the overlapped text. The neighbor system is inherited from a coarsening procedure and the prior and likelihood of the MRF is learned based on a distance metric. Experimental results show that the proposed method can achieve 87.97% recall for handwriting and 91.44% recall for machine printed text.

In this paper, we describe an approach to segment handwritten text, machine printed text and noise from annotated machine printed documents. Three categories of word level features are extracted. We use a modified K-Means clustering algorithm for classification followed by a relabeling procedure using Markov Random Field(MRF) based on a concept of neighboring patches and Belief Propagation(BP) rules. Experimental results on an imbalanced data set show that our approach achieves an overall recall of 96.33%.

In this paper, we present a new text line extraction method for handwritten Arabic documents. The proposed technique is based on a generalized adaptive local connectivity map (ALCM) using a steerable directional filter. The algorithm is designed to solve the particularly complex problems seen in handwritten documents such as fluctuating, touching or crossing text lines. The proposed algorithm consists of three steps. Firstly, a steerable filter is used to probe and determine foreground intensity along multiple directions at each pixel while generating the ALCM. The ALCM is then binarized using an adaptive thresholding algorithm to get a rough estimate of the location of the text lines. In the second step, connected component analysis is used to classify text and non text patterns in the generated ALCM to refine the location of the text lines. Finally, the text lines are separated by superimposing the text line patterns in the ALCM on the original document image and extracting the connected components covered by the pattern mask. Analysis of experimental results on the DARPA MADCAT Arabic handwritten document data indicate that the method is robust and is capable of correctly isolating handwritten text lines even on challenging document images.

OCR of Devanagari script presents a wide range of challenges that are not seen in Latin based scripts. This paper outlines the implementation of a neural network based Devanagari OCR. Experimental results on a standard data set are reported and analyzed.

Traditional handwriting recognition algorithms rely heavily on small lexicons and clean word images. Unfortunately, emergency medical documents do not satisfy either of these conditions. This is a significant road-block that is hampering efforts to rapidly convert valuable offline healthcare handwriting data into digital content that can be efficiently mined for information. This paper describes a strategy whereby given an image representing a noisy handwritten word from a medical document, and a large lexicon consisting of English, medical and pharmacological words, symbols, abbreviations and acronyms, significantly reduces the size of the lexicon while keeping the unknown desired entry within the lexicon. The approach combines geometric interpretations of the word image along with contextual inference of concepts to reduce lexicons for word recognition. The data extracted can then be efficiently and securely disseminated for epidemiological and outbreak detection/analysis. Experimental results on NY State PCR forms are reported.

This paper presents an algorithm using adaptive local connectivity map for retrieving text lines from the complex handwritten documents such as handwritten historical manuscripts. The algorithm is designed for solving the particularly complex problems seen in handwritten documents. These problems include fluctuating text lines, touching or crossing text lines and low quality image that do not lend themselves easily to binarizations. The algorithm is based on connectivity features similar to local projection profiles, which can be directly extracted from gray scale images. The proposed technique is robust and has been tested on a set of complex historical handwritten documents such as Newton's and Galileo's manuscripts. A preliminary testing shows a successful location rate of above 95% for the test set.

In this paper we present a top-down, projection-profile based algorithm to separate text blocks from image blocks in a Devanagari document. We use a distinctive feature of Devanagari text, called Shirorekha (Header Line) to analyze the pattern produced by Devanagari text in the horizontal profile. The horizontal profile corresponding to a text block possesses certain regularity in frequency, orientation and shows spatial cohesion. The algorithm uses these features to identify text blocks in a document image containing both text and graphics.

This paper describes the issues involved in the design of a system for evaluating improvements in the performance of a real-time address recognition system being used by the United States Postal Service for processing mail-piece images. Evaluation of the performance of recognition systems is normally carried out by measuring the performance of the system on a representative sample of images. Designing a comprehensive and valid testing scenario is a complex task that requires careful attention. Sampling live mail-stream to generate a deck of images representative of the general mail-stream for testing, truthing (generating reference data on a significant number of images), grading and evaluation, and designing tools to facilitate these functions are important topics that need to be addressed. This paper describes the efforts of the United States Postal Service and CEDAR towards developing an infrastructure for sampling, truthing and testing of mail-stream images.