We present a novel method for detecting text lines in historical handwritten and printed document images. Our hybrid technique begins by leveraging deep neural networks to perform multi-class pixel-wise prediction. The predictor not only discovers text and graphics pixels in the document, but it is also designed to automatically adhere contiguous regions from the same text line while also predicting buffers that prevent disassociated text lines from merging. The system breaks neural "ties" through dynamic programming. To the best of our knowledge, our system is the first neural system to predict the entire perimeters of full text lines. Also, to aid in scaling and full-scope awareness, the network during training is initially given small regions of the image to study and then expands its scope to full images as training continues. Our goal for line segmentation is to enable automatic transcription on huge heterogenous collections of historical images, so we use transcription accuracy as our metric. We document and leverage our state-of-the-art transcription system as an evaluation harness for scoring our segmenter along with various other competitor segmenters. We then show the effectiveness of our system as it relates to other systems by comparing it to both known data sets (IAM) and to three 50K-word "in the wild" test sets consisting of (a) US handwritten wills and deeds, (b) US historical newsprint images, and (c) Spanish Church and Government records.

The FamilySearch Indexing project has enabled hundreds of thousands of volunteers to transcribe billions of records, making it one of the largest crowdsourcing initiatives in the world. Assuring high quality transcriptions (i.e., indexes) with a reasonable amount of volunteer effort is essential to keep pace with the mounds of newly digitized documents. Using historical data, we show the relationship between prior experience and native language on transcriber agreement. We then present a field experiment comparing the effectiveness (accuracy) and efficiency (time) of two quality control mechanisms: (1) Arbitration -- the existing mechanism wherein two volunteers independently transcribe records and disagreements go to an arbitrator, and (2) Peer Review -- a mechanism wherein one volunteer's work is reviewed by another volunteer. Peer Review is significantly more efficient, though not as effective for certain fields as Arbitration. Design suggestions for FamilySearch Indexing and related crowdsourcing initiatives are provided.

Relation extraction has been a subject of significant study over the past decade. Most relation extractors have been developed by combining the training of complex computational systems on large volumes of annotations with extensive rule writing by language experts. Moreover, many relation extractors are reliant on other non-trivial NLP technologies which themselves are developed through significant human efforts, such as entity tagging, parsing, etc. Due to the high cost of creating and assembling the required resources, relation extractors have typically been developed for only high-resourced languages. In this paper, we describe a near-zero-cost methodology to build relation extractors for significantly distinct non-English languages using only freely available Wikipedia and other web documents, and some knowledge of English. We apply our methodology and build alma-mater, birthplace, father, occupation, and spouse relation extractors in Greek, Spanish, Russian, and Chinese. We conduct evaluations of induced relations at the file level which are the most refined we have seen in the literature.

Although Arabic is currently one of the most widely spoken languages in the world, there has been relatively little speech recognition research on Arabic compared to other languages. Moreover, most previous work has concentrated on the recognition of formal rather than dialectal Arabic. This paper reports on our project at the 2002 Johns Hopkins Summer Workshop, which focused on the recognition of dialectal Arabic. Three problems were addressed: (a) the lack of short vowels and other pronunciation information in Arabic texts; (b) the morphological complexity of Arabic; and (c) the discrepancies between dialectal and formal Arabic. We present novel approaches to automatic vowel restoration, morphology-based language modeling and the integration of out-of-corpus language model data, and report significant word error rate improvements on the LDC Arabic CallHome task.

MEI (Mandarin-English Information) is an English-Chinese crosslingual spoken document retrieval (CL-SDR) system developed during the Johns Hopkins University Summer Workshop 2000. We integrate speech recognition, machine translation, and information retrieval technologies to perform CL-SDR. MEI advocates a multi-scale paradigm, where both Chinese words and subwords (characters and syllables) are used in retrieval. The use of subword units can complement the word unit in handling the problems of Chinese word tokenization ambiguity, Chinese homophone ambiguity, and out-of-vocabulary words in audio indexing. This paper focuses on multi-scale audio indexing in MEI. Experiments are based on the Topic Detection and Tracking Corpora (TDT-2 and TDT-3), where we indexed Voice of America Mandarin news broadcasts by speech recognition on both the word and subword scales. We discuss the development of the MEI syllable recognizer, the representations of spoken documents using overlapping subword n-grams and lattice structures. Results show that augmenting words with subwords is beneficial to CL-SDR performance.

This paper presents a multi-scale retrieval approach in MEI (Mandarin-English Information), an English-Chinese crosslingual spoken document retrieval (CL-SDR) system. It accepts an entire English news story (from newspaper text) as the input query, and automatically retrieves Mandarin news stories (from broadcast audio). This allows the user to search for personally relevant content across the language and media barriers – a cross-lingual and cross-media retrieval task. MEI advocates a multi-scale paradigm for the retrieval task. Multiscale refers to the use of both words and subwords (Chinese characters and syllables) for retrieval. Words offer lexical knowledge to enhance precision, and subwords can potentially alleviate some prevailing problems in CL-SDR, e.g. open vocabularies in translation and recognition, out-of-vocabulary words in audio indexing, and ambiguities in Chinese homophones and word tokenizaiton. We present techniques for word-subword fusion, which improved retrieval performance in our experiments with the Topic Detection and Tracking collection.

An automatic technique which attempts to determine and cluster topical information in text and transcribed speech is described. The method uses an electronic dictionary which is recursively closed. The method identifies each word of a given excerpt of text by its root form in the dictionary and a salience is affixed. The salience from each individual word is then propagated to those terms which define it, which terms may themselves be propagated. Thus, each word in the excerpt spawns a directed graph of dictionary words and related weights. A combining function is then applied to the union of these graphs which enhances any mutual information. The N words with the largest combined scores are identified as either the topic or words pertaining to the topic. These top N words and their scores can then be used in cross-message correlations as a means to cluster topics.