Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated imagecaption data in other languages; in the long run, this might widen inequitable access to TTI technology.In this work, we thus investigate multilingual TTI (termed mTTI) and the current potential of neural machine translation (NMT) to bootstrap mTTI systems.We provide two key contributions.1) Relying on a multilingual multi-modal encoder, we provide a systematic empirical study of standard methods used in cross-lingual NLP when applied to mTTI: TRANSLATE TRAIN, TRANS-LATE TEST, and ZERO-SHOT TRANSFER.2) We propose Ensemble Adapter (ENSAD), a novel parameter-efficient approach that learns to weigh and consolidate the multilingual text knowledge within the mTTI framework, mitigating the language gap and thus improving mTTI performance.Our evaluations on standard mTTI datasets COCO-CN, Multi30K Task2, and LAION-5B demonstrate the potential of translation-enhanced mTTI systems and also validate the benefits of the proposed EN-SAD which derives consistent gains across all datasets.Further investigations on model variants, ablation studies, and qualitative analyses provide additional insights on the inner workings of the proposed mTTI approaches.

We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system. Though we train using 70% spoken-form data, our teacher models perform comparably to XLM-R and mT5 when evaluated on the written-form Cross-lingual Natural Language Inference (XNLI) corpus. We perform a second stage of pretraining on our teacher models using in-domain data from our system, improving error rates by 3.86% relative for intent classification and 7.01% relative for slot filling. We find that even a 170M-parameter model distilled from our Stage 2 teacher model has 2.88% better intent classification and 7.69% better slot filling error rates when compared to the 2.3B-parameter teacher trained only on public data (Stage 1), emphasizing the importance of in-domain data for pretraining. When evaluated offline using labeled NLU data, our 17M-parameter Stage 2 distilled model outperforms both XLM-R Base (85M params) and DistillBERT (42M params) by 4.23% to 6.14%, respectively. Finally, we present results from a full virtual assistant experimentation platform, where we find that models trained using our pretraining and distillation pipeline outperform models distilled from 85M-parameter teachers by 3.74%-4.91% on an automatic measurement of full-system user dissatisfaction.

Semantic parsing is one of the key components of natural language understanding systems.A successful parse transforms an input utterance to an action that is easily understood by the system.Many algorithms have been proposed to solve this problem, from conventional rulebased or statistical slot-filling systems to shiftreduce based neural parsers.For complex parsing tasks, the state-of-the-art method is based on autoregressive sequence to sequence models to generate the parse directly.This model is slow at inference time, generating parses in O(n) decoding steps (n is the length of the target sequence).In addition, we demonstrate that this method performs poorly in zero-shot cross-lingual transfer learning settings.In this paper, we propose a non-autoregressive parser which is based on the insertion transformer to overcome these two issues.Our approach 1) speeds up decoding by 3x while outperforming the autoregressive model and 2) significantly improves cross-lingual transfer in the low-resource setting by 37% compared to autoregressive baseline.We test our approach on three well-known monolingual datasets: ATIS, SNIPS and TOP.For cross lingual semantic parsing, we use the MultiATIS++ and the multilingual TOP datasets.

Synthetic data generation for optical character recognition (OCR) promises unlimited training data at zero annotation cost. With enough fonts and seed text, we should be able to generate data to train a model that approaches or exceeds the performance with real annotated data. Unfortunately, this is not always the reality. Unconstrained image settings, such as internet memes, scanned web pages, or newspapers, present diverse scripts, fonts, layouts, and complex backgrounds, which cause models trained with synthetic data to break down. In this work, we investigate the synthetic image generation problem on a large multilingual set of unconstrained document images. Our work presents a comprehensive evaluation of the impact of synthetic data attributes on model performance. The results provide a recipe for synthetic data generation that will help guide future research.

We propose a method designed to push the frontiers of unconstrained face recognition in the wild with an emphasis on extreme out-of-plane pose variations. Existing methods either expect a single model to learn pose invariance by training on massive amounts of data or else normalize images by aligning faces to a single frontal pose. Contrary to these, our method is designed to explicitly tackle pose variations. Our proposed Pose-Aware Models (PAM) process a face image using several pose-specific, deep convolutional neural networks (CNN). 3D rendering is used to synthesize multiple face poses from input images to both train these models and to provide additional robustness to pose variations at test time. Our paper presents an extensive analysis of the IARPA Janus Benchmark A (IJB-A), evaluating the effects that landmark detection accuracy, CNN layer selection, and pose model selection all have on the performance of the recognition pipeline. It further provides comparative evaluations on IJB-A and the PIPA dataset. These tests show that our approach outperforms existing methods, even surprisingly matching the accuracy of methods that were specifically fine-tuned to the target dataset. Parts of this work previously appeared in [1] and [2].

We propose EPAT, (Euclidean Perturbation Analysis and Transform) a novel unsupervised adaptation approach for improving the accuracy of any facial landmark detector by characterizing the stability of landmark prediction on test images. In EPAT, a test image is transformed several times using a set of Euclidean transforms, producing several perturbed images. The black box landmark detector is used to find facial landmarks on each perturbed version of the test image. Subsequently, inverse transforms are applied to the corresponding landmarks in order to map them back to the original image. Mean and variance are calculated for all inversely transformed detection. Mean and variance represent the new ensemble prediction and the sensitivity of the underlying landmark detector, respectively. We also introduce affine variance (AV) of facial landmarks. AV is used as a measure of the stability of the predicted landmarks and a criterion for selecting a good data adaptation model which effectively addresses potential mismatches between test and training data of the underlying landmark detector. EPAT is evaluated using four state-of-the-art landmark detectors on the standard 300W dataset and also incorporated into a face recognition pipeline to show improved recognition accuracy on the challenging IJB-A dataset.

This paper presents a system for transcribing 1990 US census forms. Extraction of information from census forms is useful for creating a genealogy database and better archiving census forms. We trained CTC/LSTM-RNN networks as our OCR engine. We solved the major challenge in language modeling by defining syntactical constraints with WFST language models. We made two major technical contributions in this paper. Firstly, 1990 US census forms were automatically transcribed with compelling accuracy for the first time using our system, which can be useful in downstream study in information extracted from census forms. Secondly, we designed a novel post-processing algorithm that improved the recognition accuracy of surnames significantly.

We present a novel end-to-end trainable OCR system combining a CNN for feature extraction with 1-D LSTMs for sequence modeling. We present results on English and Arabic handwriting data, and on English machine print data, showing state-of-the-art performance. We believe that our method is simpler than existing 2D LSTM models, and will make it easier to use techniques borrowed from CNN research in computer vision to improve OCR performance.

We propose a method to push the frontiers of unconstrained face recognition in the wild, focusing on the problem of extreme pose variations. As opposed to current techniques which either expect a single model to learn pose invariance through massive amounts of training data, or which normalize images to a single frontal pose, our method explicitly tackles pose variation by using multiple pose specific models and rendered face images. We leverage deep Convolutional Neural Networks (CNNs) to learn discriminative representations we call Pose-Aware Models (PAMs) using 500K images from the CASIA WebFace dataset. We present a comparative evaluation on the new IARPA Janus Benchmark A (IJB-A) and PIPA datasets. On these datasets PAMs achieve remarkably better performance than commercial products and surprisingly also outperform methods that are specifically fine-tuned on the target dataset.

We present a fully trainable solution for binarization of degraded document images using extremely randomized trees. Unlike previous attempts that often use simple features, our method encodes all heuristics about whether or not a pixel is foreground text into a high-dimensional feature vector and learns a more complicated decision function. We introduce two novel features, the Logarithm Intensity Percentile (LIP) and the Relative Darkness Index (RDI), and combine them with low level features, and reformulated features from existing binarization methods. Experimental results show that using small sample size (about 1.5% of all available training data), we can achieve a binarization performance comparable to manually-tuned, state-of-the-art methods. Additionally, the trained document binarization classifier shows good generalization capabilities on out-of-domain data.

Classically, face recognition depends on computing the similarity (or distance) between a pair of face images and/or their respective representations, where each subject is represented by one image. Template-based face recognition was introduced by the release of IARPA's Janus Benchmark-A (IJB-A) dataset, in which each enrolled subject is represented by a group of one or more images, called a template. The group of images comprising a template might have been acquired using different head poses, illuminations, ages and facial expressions. Template images could come from still images or video frames. Therefore, measuring the similarity between templates representing two subjects significantly increases the number of pairwise image comparisons (i.e., O(NM), where N and M are the number of image templates being compared). As the number of enrolled subjects, K, increases, both computational and space requirements become computationally prohibitive. To address this challenge, we present a novel approximate nearest-neighbor (ANN) search-based solution. Given a query template, ANN methods are used to find similar face images. Retrieved images are used to construct a template pool that is used to find the correct identity of the query subject. The proposed approach largely reduces the number of imposter template-pair comparisons. Experimental results on the IJB-A dataset show that the proposed approach achieves significant speed-up and storage savings, without sacrificing accuracy.

We introduce our method and system for face recognition using multiple pose-aware deep learning models. In our representation, a face image is processed by several pose-specific deep convolutional neural network (CNN) models to generate multiple pose-specific features. 3D rendering is used to generate multiple face poses from the input image. Sensitivity of the recognition system to pose variations is reduced since we use an ensemble of pose-specific CNN features. The paper presents extensive experimental results on the effect of landmark detection, CNN layer selection and pose model selection on the performance of the recognition pipeline. Our novel representation achieves better results than the state-of-the-art on IARPA's CS2 and NIST's IJB-A in both verification and identification (i.e. search) tasks.

In this paper, we introduce a new operational platform for end-to-end document image analysis, recognition, and machine translation. The Raytheon BBN Document Analysis Service (BBN DAS) performs the following operations on scanned machine-print document images: (1) image pre-processing and segmentation to identify homogenous zones of text, (2) text recognition to convert the text zones into electronic text, (3) machine translation for converting the text from the native language of the document into English, and (4) document archiving and indexing for effective content-based search. BBN DAS uses a service-oriented architecture (SOA), which offers modularity and scalability for operation on hardware configurations ranging from a laptop to distributed multi-node server environments. This paper describes the platform architecture, the process of configuring it for Arabic newsprint documents and resulting performance results of the Arabic system.