In this paper, we address the problem of learning to perform sequential OCR on photos of street name signs in a language for which no labeled data exists. Our approach leverages easily-generated synthetic data and existing labeled data in other languages to achieve reasonable performance on these unlabeled images, through a combination of a novel domain adaptation technique based on gradient reversal and a multi-task learning scheme. In order to accomplish this, we introduce and release two new datasets - Hebrew Street Name Signs (HSNS) and Synthetic Hebrew Street Name Signs (SynHSNS) - while also making use of the existing French Street Name Signs (FSNS) dataset. We demonstrate that by using a synthetic dataset of Hebrew characters and a labeled dataset of French street name signs in natural images, it is possible to achieve a significant improvement on real Hebrew street name sign transcription, where the synthetic Hebrew data and real French data each overlap with different features of the images we wish to transcribe.

We present a neural network model — based on Convolutional Neural Networks, Recurrent Neural Networks and a novel attention mechanism — which achieves 84.2% accuracy on the challenging French Street Name Signs (FSNS) dataset, significantly outperforming the previous state of the art (Smith'16), which achieved 72.46%. Furthermore, our new method is much simpler and more general than the previous approach. To demonstrate the generality of our model, we show that it also performs well on an even more challenging dataset derived from Google Street View, in which the goal is to extract business names from store fronts. Finally, we study the speed/accuracy tradeoff that results from using CNN feature extractors of different depths. Surprisingly, we find that deeper is not always better (in terms of accuracy, as well as speed). Our resulting model is simple, accurate and fast, allowing it to be used at scale on a variety of challenging real-world text extraction problems.

In order to cope with the vast diversity of book content and typefaces, it is important for OCR systems to leverage the strong consistency within a book but adapt to variations across books. We describe a system that combines two parallel correction paths using document-specific image and language models. Each model adapts to shapes and vocabularies within a book to identify inconsistencies as correction hypotheses, but relies on the other for effective cross-validation. Using the open source Tesseract engine as baseline, results on a large data set of scanned books demonstrate that word error rates can be reduced by 25 percent using this approach.

A novel system is described that significantly enhances the usefulness of handwritten notes taken during a presentation by creating a multimedia document that includes scanned images of handouts, personal notes, and links to a multimedia recording of the presentation. Notes are linked to the e-presentation media with automatic content analysis without any special notes capture device. Layout segmentation and template matching automatically detects the presence of presentation handouts during scanning. Presentation-level and slide-level linking of handouts to e-media use text and image features from slides. Experimental results show 95% accuracy in linking of the scanned handouts to the e-presentation media.

Overcoming the semantic-feature gap and adapting to context are two main challenges in content-based retrieval. The problem is even more complicated for unstructured videos such as automated recordings of meetings. To address this problem, we propose a model-based approach to meeting retrieval with user controlled weighting for dynamic similarity comparison. Each video is represented by an HMM, and the similarity between videos is determined by comparing the corresponding models. Users can control the relative importance of temporal and static features by adjusting a weighting parameter in a way similar to content-based image retrieval. Experimental results demonstrate the feasibility and versatility of this approach.

Adaptive Gaussian mixtures have been used for modeling nonstationary temporal distributions of pixels in video surveillance applications. However, a common problem for this approach is balancing between model convergence speed and stability. This paper proposes an effective scheme to improve the convergence rate without compromising model stability. This is achieved by replacing the global, static retention factor with an adaptive learning rate calculated for each Gaussian at every frame. Significant improvements are shown on both synthetic and real video data. Incorporating this algorithm into a statistical framework for background subtraction leads to an improved segmentation performance compared to a standard method.

A multimedia note taking system called MinuteAid is described that lets users of laptops include multimedia objects in notes they write during a presentation. The multimedia data is provided by devices that capture slide images, whiteboard images, audio and video clips. The MinuteAid system includes a software interface between these devices and a commonly available word processing application (Microsoft Word) and lets the user either insert copies of multimedia objects while they're captured or references to multimedia objects that are automatically retrieved and inserted in the document later. We describe the architecture, implementation and user experience with this system.

The components of a key frame selection algorithm for a paper-based multimedia browsing interface called Video Paper are described. Analysis of video image frames is combined with the results of processing the closed caption to select key frames that are printed on a paper document together with the closed caption. Bar codes positioned near the key frames allow a user to play the video from the corresponding times. This paper describes several component techniques that are being investigated for key frame selection in the Video Paper system, including face detection and text recognition. The Video Paper system implementation is also discussed.

Background subtraction is an essential processing component for many video applications. However, its development has largely been application driven and done in an ad hoc manner. In this paper, we provide a Bayesian formulation of background segmentation based on Gaussian mixture models. We show that the problem consists of two density estimation problems, one application independent and one application dependent, and a set of intuitive and theoretically optimal solutions can be derived for both. The proposed framework was tested on meeting and traffic videos and compared favorably to other well-known algorithms.

Recorded meetings are useful only if people can find, access, and browse them easily. Key-frames and video skims are useful representations that can enable quick previewing of the content without actually watching a meeting recording from beginning to end. This paper proposes a new method for creating meeting video skims based on audio and visual activity analysis together with text analysis. Audio activity analysis is performed by analyzing sound directions-indicating different speakers-and audio amplitude. Detection of important visual events in a meeting is achieved by analyzing the localized luminance variations in consideration with the omni-directional property of the video captured by our meeting recording system. Text analysis is based on the term frequency-inverse document frequency measure. The resulting video skims better capture the important meeting content compared to the skims obtained by uniform sampling.

An algorithm is presented that automatically matches images of presentation slides to the symbolic source file (e.g., PowerPoint™ or Acrobat™) from which they were generated. The images are captured either by tapping the video output from a laptop connected to a projector or by taking a picture of what's displayed on the screen in a conference room. The matching algorithm extracts features from the image data, including OCR output, edges, projection profiles, and layout and determines the symbolic file that contains the most similar collection of features. This algorithm enables several unique applications for enhancing a meeting in real-time and accessing the audio and video that were recorded while a presentation was being given. These applications include the simultaneous translation of presentation slides during a meeting, linking video clips inside a PowerPoint file that show how each slide was described by the presenter, and retrieving presentation recordings using digital camera images as queries.

Video Paper is a prototype system for multimedia browsing, analysis, and replay. Key frames extracted from a video recording are printed on paper together with bar codes that allow for random access and replay. A transcript for the audio track can also be shown so that users can read what was said, thus making the document a stand-alone representation for the contents of the multimedia recording. The Video Paper system has been used for several applications, including the analysis of recorded meetings, broadcast news, oral histories and personal recordings. This demonstration will show how the Video Paper system was applied to these domains and the various replay systems that were developed, including a self-contained portable implementation on a PDA and a fixed implementation on desktop PC.

The design and implementation of a portable meeting recorder is presented. Composed of an omni-directional video camera with four-channel audio capture, the system saves a view of all the activity in a meeting and the directions from which people spoke. Subsequent analysis computes metadata that includes video activity analysis of the compressed data stream and audio processing that helps locate events that occurred during the meeting. Automatic calculation of the room in which the meeting occurred allows for efficient navigation of a collection of recorded meetings. A user interface is populated from the metadata description to allow for simple browsing and location of significant events.

It has been shown that simple substitution ciphers can be solved using statistical methods such as probabilistic relaxation. However, the utility of such solutions has been limited by their inability to cope with noise encountered in practical applications. We propose a new solution to substitution deciphering based on hidden Markov models. We show that our algorithm is more accurate than relaxation and much more robust in the presence of noise, making it useful for applications in compressed document processing. Recovering character interpretations from the sequence of cluster identifiers in a symbolically compressed document can be treated as a cipher problem. Although a significant amount of noise is present in the cluster sequence, enough information can be recovered with a robust deciphering algorithm to accomplish certain document analysis tasks. The feasibility of this approach is demonstrated in a multilingual document duplicate detection system.

The ability to automatically record the marks applied to paper documents on their electronic originals would preserve the information represented by those annotations. Users could even lose the original paper document. The marked-up version could be regenerated by merely reprinting it. We describe a solution that saves an electronic representation for the highlights users commonly apply over the top of machine-printed text. A unique combination of algorithms is presented that maps the image captured from a pen scanner affixed to a highlighting pen onto text strings in electronic documents. Documents are automatically located in a large database using characteristics of the highlighted text. We describe here the system components, including the image recognition algorithms, and discuss their performance in finding a unique mapping from an image of text onto a sequence of words in an electronic document within a large database.

A new family of symbolic compression algorithms has recently been developed that includes the ongoing JBIG2 standardization effort as well as related commercial products. These techniques are specifically designed for binary document images. They cluster individual blobs in a document and store the sequence of occurrence of blobs and representative blob templates, hence the name symbolic compression. This paper describes a method for duplicate detection on symbolically compressed document images. It recognizes the text in an image by deciphering the sequence of occurrence of blobs in the compressed representation. We propose a Hidden Markov Model (HMM) method for solving such deciphering problems and suggest applications in multilingual document duplicate detection.

this report, combine decisions (or statistics) from all the text lines on the page to support a final decision at the page level. The order of execution is as follows. Within a page, we visit each text line (in arbitrary order), extracting features and computing statistics for each (details of these will be given below). Each text line is immediately classified as Asian or Latin (or rejected), in a way that requires no prior knowledge of the orientation. Feature statistics which support other decisions are accumulated and, at the end of the page, the remaining decisions are made in the following order. First, we distinguish between Asian and Latin scripts, determined by a majority vote of the text-line classifications already decided. If the page's script is Asian, we then decide the page's language using feature statistics accumulated from all text lines (again, at any orientation); then, using knowledge of the Asian-script page's language, we detect the page's orientation. If the page's script is Latin, we detect the page's orientation first and then decide the page's language. An overview of this process is shown in Figure 1. Details of the methods used for each of these decisions, along with test results, are discussed in the sections that follow. But first, we briefly summarize the training and test data used. 3 Training and Testing Data

The reading of names and addresses is one of the most complex tasks in automated forms processing. This paper describes an integrated real-time system to read names and addresses on tax forms of the U.S. Internal Revenue Service. The Name and Address Block Reader (NABR) system accepts both machine-printed and hand-printed address block images as input. The application software has two major steps: document analysis (connected component analysis, address block extraction, label detection, hand-print/machine-print discrimination) and document recognition. Document recognition has two nonidentical streams for machine-print and hand-print: the key steps are address parsing, character recognition, word recognition, and postal database lookup. (ZIP+4 and City-State-ZIP files.) System output is a packet containing the results of recognition together with database access status file. Real-time throughput (8500 forms/h) is achieved by employing a loosely coupled multiprocessing architecture where successive input images are distributed to available address recognition processors. The functional architecture, software design, system architecture, and the hardware implementation are described. Performance evaluation on machine-printed and handwritten addresses are presented.

The paper examines the general classifier combination problem under strict separation of the classifier and combinator design. Several desirable combinator properties are identified: omnitype mixed type and correlated classifier combination, redundant classifier elimination, model complexity control, and dynamic selection combination. By adapting some of the theories and algorithms developed for neural network learning. They present a combination model which provides a solution to these problems. Experimental results on handwritten digits verify these findings.

Commercial forms-reading systems for extraction of data from forms do not meet acceptable accuracy requirements on forms filled out by hand. In December 1993, NIST called industry and research organizations working in the area of handwriting recognition to participate in a test to determine the state of the art in the area. A database of form images containing actual responses received by the US Census Bureau was provided. The handwritten responses are very loosely constrained in terms of writing style, format of response and choice of text. The sizes of the lexicons provided are very large (about 50000 entries) and yet the coverage is incomplete (about 70%). In this paper we discuss the approach taken by CEDAR to automate the task of reading the census forms. The subtasks of field extraction and phrase recognition are described.

The reading of names and addresses is one of the most complex tasks in automated forms processing. The paper describes an integrated real time system to read names and addresses on tax forms of the Internal Revenue Service of the United States. The Name and Address Block Reader (NABR) system accepts both machine printed and hand printed address block images as input. The application software has two major steps: document analysis (connected component analysis, address block extraction, label detection, hand print/machine print discrimination); and document recognition. Document recognition has two non identical streams for machine print and hand print; key steps are: address parsing, character recognition, word recognition and postal database lookup (ZIP+4 and City-State-ZIP files). Real time throughput (8,500 forms per hour) is achieved by employing a loosely coupled multiprocessing architecture. The functional architecture, software design, system architecture and hardware implementation are described. Performance evaluation on machine printed and handwritten addresses are presented.

Image size normalization is a crucial preprocessing stage in the development of robust object recognizers. A new method of image size normalization based on multirate filter theory is proposed. Comparisons with ratio-based normalization and simple scaling are made. The effect of each normalization method on handwritten digit recognition is evaluated. Recognition incorporates global and local features extracted from normalized digit images and used with a neural network and K-nearest neighbor classifier performance evaluation is based on recognition accuracy, reject versus error graph, figure of merit and processing time required by each method. Multirate-based normalization yields better recognition performance at the cost of increased computation, whereas ratio-based normalization and simple scaling method require less processing time with reduced recognition performance.