In this study, we investigate robustness against covariate drift in spoken language understanding (SLU).Covariate drift can occur in SLU when there is a drift between training and testing regarding what users request or how they request it.To study this we propose a method that exploits natural variations in data to create a covariate drift in SLU datasets.Experiments show that a state-of-the-art BERT-based model suffers performance loss under this drift.To mitigate the performance loss, we investigate distributionally robust optimization (DRO) for finetuning BERT-based models.We discuss some recent DRO methods, propose two new variants and empirically show that DRO improves robustness under drift.

We investigate post-OCR correction in a setting where we have access to different OCR views of the same document.The goal of this study is to understand if a pretrained language model (LM) can be used in an unsupervised way to reconcile the different OCR views such that their combination contains fewer errors than each individual view.This approach is motivated by scenarios in which unconstrained text generation for error correction is too risky.We evaluated different pretrained LMs on two datasets and found significant gains in realistic scenarios with up to 15% WER improvement over the best OCR view.We also show the importance of domain adaptation for post-OCR correction on out-of-domain documents.

Open Information Extraction systems extract ("subject text", "relation text", "object text") triples from raw text.Some triples are textual versions of facts, i.e., non-canonicalized mentions of entities and relations.In this paper, we investigate whether it is possible to infer new facts directly from the open knowledge graph without any canonicalization or any supervision from curated knowledge.For this purpose, we propose the open link prediction task, i.e., predicting test facts by completing ("subject text", "relation text", ?) questions.An evaluation in such a setup raises the question if a correct prediction is actually a new fact that was induced by reasoning over the open knowledge graph or if it can be trivially explained.For example, facts can appear in different paraphrased textual variants, which can lead to test leakage.To this end, we propose an evaluation protocol and a methodology for creating the open link prediction benchmark OLPBENCH.We performed experiments with a prototypical knowledge graph embedding model for open link prediction.While the task is very challenging, our results suggests that it is possible to predict genuinely new facts, which can not be trivially explained.

A typical architecture for end-to-end entity linking systems consists of three steps: mention detection, candidate generation and entity disambiguation.In this study we investigate the following questions: (a) Can all those steps be learned jointly with a model for contextualized text-representations, i.e.BERT (Devlin et al., 2019)?(b) How much entity knowledge is already contained in pretrained BERT?(c) Does additional entity knowledge improve BERT's performance in downstream tasks?To this end, we propose an extreme simplification of the entity linking setup that works surprisingly well: simply cast it as a per token classification over the entire entity vocabulary (over 700K classes in our case).We show on an entity linking benchmark that (i) this model improves the entity representations over plain BERT, (ii) that it outperforms entity linking architectures that optimize the tasks separately and (iii) that it only comes second to the current state-of-the-art that does mention detection and entity disambiguation jointly.Additionally, we investigate the usefulness of entity-aware token-representations in the text-understanding benchmark GLUE, as well as the question answering benchmarks SQUAD V2 and SWAG and also the EN-DE WMT14 machine translation benchmark.To our surprise, we find that most of those benchmarks do not benefit from additional entity knowledge, except for a task with very small training data, the RTE task in GLUE, which improves by 2%.

Financial risk, defined as the chance to deviate from return expectations, is most commonly measured with volatility. Due to its value for investment decision making, volatility prediction is probably among the most important tasks in finance and risk management. Although evidence exists that enriching purely financial models with natural language information can improve predictions of volatility, this task is still comparably underexplored. We introduce PRoFET, the first neural model for volatility prediction jointly exploiting both semantic language representations and a comprehensive set of financial features. As language data, we use transcripts from quarterly recurring events, so-called "earnings calls"; in these calls, the performance of publicly traded companies is summarized and prognosticated by their management. We show that our proposed architecture, which models verbal context with an attention mechanism, significantly outperforms the previous state-of-the-art and other strong baselines. Finally, we visualize this attention mechanism on the token-level, thus aiding interpretability and providing a use case of PRoFET as a tool for investment decision support.