Fixed-size learned representations (dense representations, or embeddings) are widely used in many machine learning applications across language, vision or speech modalities. This paper investigates the role of the temperature parameter in contrastive training for text embeddings. We shed light on the impact this parameter has on the intrinsic dimensionality of the embedding spaces obtained, and show that lower intrinsic dimensionality is further correlated with effective compression of embeddings. We still observe a trade-off between absolute performance and effective compression and we propose temperature aggregation methods which reduce embedding size by an order of magnitude with minimal impact on quality.

Anna Currey, Maria Nadejde, Raghavendra Reddy Pappagari, Mia Mayer, Stanislas Lauly, Xing Niu, Benjamin Hsu, Georgiana Dinu. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Targeted evaluations have found that machine translation systems often output incorrect gender in translations, even when the gender is clear from context.Furthermore, these incorrectly gendered translations have the potential to reflect or amplify social biases.We propose gender-filtered self-training (GFST) to improve gender translation accuracy on unambiguously gendered inputs.Our GFST approach uses a source monolingual corpus and an initial model to generate gender-specific pseudo-parallel corpora which are then filtered and added to the training data.We evaluate GFST on translation from English into five languages, finding that it improves gender accuracy without damaging generic quality.We also show the viability of GFST on several experimental settings, including re-training from scratch, fine-tuning, controlling the gender balance of the data, forward translation, and back-translation. 1

Neural Machine Translation (NMT) models are sensitive to small perturbations in the input.Robustness to such perturbations is typically measured using translation quality metrics such as BLEU on the noisy input.This paper proposes additional metrics which measure the relative degradation and changes in translation when small perturbations are added to the input.We focus on a class of models employing subword regularization to address robustness and perform extensive evaluations of these models using the robustness measures proposed.Results show that our proposed metrics reveal a clear trend of improved robustness to perturbations when subword regularization methods are used.

Neural machine translation achieves impressive results in high-resource conditions, but performance often suffers when the input domain is low-resource.The standard practice of adapting a separate model for each domain of interest does not scale well in practice from both a quality perspective (brittleness under domain shift) as well as a cost perspective (added maintenance and inference complexity).In this paper, we propose a framework for training a single multi-domain neural machine translation model that is able to translate several domains without increasing inference time or memory usage.We show that this model can improve translation on both highand low-resource domains over strong multidomain baselines.In addition, our proposed model is effective when domain labels are unknown during training, as well as robust under noisy data conditions.

This paper proposes a novel method to inject custom terminology into neural machine translation at run time.Previous works have mainly proposed modifications to the decoding algorithm in order to constrain the output to include run-time-provided target terms.While being effective, these constrained decoding methods add, however, significant computational overhead to the inference step, and, as we show in this paper, can be brittle when tested in realistic conditions.In this paper we approach the problem by training a neural MT system to learn how to use custom terminology when provided with the input.Comparative experiments show that our method is not only more effective than a state-of-the-art implementation of constrained decoding, but is also as fast as constraint-free decoding.

The state-of-the-art named entity recognition (NER) systems are supervised machine learning models that require large amounts of manually annotated data to achieve high accuracy.However, annotating NER data by human is expensive and time-consuming, and can be quite difficult for a new language.In this paper, we present two weakly supervised approaches for cross-lingual NER with no human annotation in a target language.The first approach is to create automatically labeled NER data for a target language via annotation projection on comparable corpora, where we develop a heuristic scheme that effectively selects goodquality projection-labeled data from noisy data.The second approach is to project distributed representations of words (word embeddings) from a target language to a source language, so that the sourcelanguage NER system can be applied to the target language without re-training.We also design two co-decoding schemes that effectively combine the outputs of the two projection-based approaches.We evaluate the performance of the proposed approaches on both in-house and open NER data for several target languages.The results show that the combined systems outperform three other weakly supervised approaches on the CoNLL data.

Angeliki Lazaridou, Georgiana Dinu, Marco Baroni. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

As automated image analysis progresses, there is increasing interest in richer linguistic annotation of pictures, with attributes of objects (e.g., furry, brown…) attracting most attention. By building on the recent “zero-shot learning” approach, and paying attention to the linguistic nature of attributes as noun modifiers, and specifically adjectives, we show that it is possible to tag images with attribute-denoting adjectives even when no training data containing the relevant annotation are available. Our approach relies on two key observations. First, objects can be seen as bundles of attributes, typically expressed as adjectival modifiers (a dog is something furry, brown, etc.), and thus a function trained to map visual representations of objects to nominal labels can implicitly learn to map attributes to adjectives. Second, objects and attributes come together in pictures (the same thing is a dog and it is brown). We can thus achieve better attribute (and object) label retrieval by treating images as “visual phrases”, and decomposing their linguistic representation into an attribute-denoting adjective and an object-denoting noun. Our approach performs comparably to a method exploiting manual attribute annotation, it out-performs various competitive alternatives in both attribute and object annotation, and it automatically constructs attribute-centric representations that significantly improve performance in supervised object recognition.

Context-predicting models (more commonly known as embeddings or neural language models) are the new kids on the distributional semantics block.Despite the buzz surrounding these models, the literature is still lacking a systematic comparison of the predictive models with classic, count-vector-based distributional semantic approaches.In this paper, we perform such an extensive evaluation, on a wide range of lexical semantics tasks and across many parameter settings.The results, to our own surprise, show that the buzz is fully justified, as the context-predicting models obtain a thorough and resounding victory against their count-based counterparts.

We introduce the problem of generation in distributional semantics: Given a distri-butional vector representing some mean-ing, how can we generate the phrase that best expresses that meaning? We mo-tivate this novel challenge on theoretical and practical grounds and propose a sim-ple data-driven approach to the estimation of generation functions. We test this in a monolingual scenario (paraphrase gen-eration) as well as in a cross-lingual set-ting (translation by synthesizing adjective-noun phrase vectors in English and gener-ating the equivalent expressions in Italian). 1

In this paper, we show that the lexical function model for composition of distributional semantic vectors can be improved by adopting a more advanced regression technique.We use the pathwise coordinate-descent optimized elastic-net regression method to estimate the composition parameters, and compare the resulting model with several recent alternative approaches in the task of composing simple intransitive sentences, adjective-noun phrases and determiner phrases.Experimental results demonstrate that the lexical function model estimated by elastic-net regression achieves better performance, and it provides good qualitative interpretability through sparsity constraints on model parameters.