As language models (LMs) increase in size and complexity, so too does their capacity to capture and recall knowledge from their pre-training data. This property is in some cases desirable, such as when applied to general-domain knowledge graph completion and relation extraction (RE) tasks. However, when extracting information about well-known entities from documents, it can be unclear whether LM-based systems perform well by actually solving the tasks, or instead by leveraging patterns about those entities learned from pre-training data. In realistic scenarios, documents may be domain-specific and unlike anything in the pre-training data, or may use well-known named entities in new ways. In the latter case, LMs need to disregard parts of their background knowledge to succeed, a capability we informally call suspension of disbelief. This paper uses a zero-shot RE setting to investigate whether masked LMs (MLMs) are capable of this. We devise a method called DocShRED to construct adversarial versions of well-known RE data sets like DocRED, by intentionally using named entities in ways that are inconsistent with common pre-training data. We find that BERT and RoBERTa exhibit near-random performance on DocShRED, even when using a zero-shot technique to incorporate the text as supporting information. We also find that both MLMs perform significantly better on DocShRED when entities’ surface forms are withheld using a novel method called entity isolation, highlighting the impact of background knowledge on the task. Finally, when we apply entity isolation to the biomedical RE data set BioRED, BERT outperforms BioBERT and PubMedBERT without fine-tuning, suggesting an overall improvement in generalizability.

This paper addresses the evaluation of pattern-based ontology design through experiments. An initial method for reuse of content ontology design patterns (Content ODPs) was used by the participants during the experiments. Hypotheses considered include the usefulness of Content ODPs for ontology developers, and we additionally study in what respects they are useful and what open issues remain. The main positive conclusions when using Content ODPs include: ontology developers perceived them as useful, ontology quality is improved, coverage of the task increases, usability is improved, and common modelling mistakes can be avoided.