Semantic similarity detection is a fundamental task in natural language understanding.Adding topic information has been useful for previous feature-engineered semantic similarity models as well as neural models for other tasks.There is currently no standard way of combining topics with pretrained contextual representations such as BERT.We propose a novel topic-informed BERT-based architecture for pairwise semantic similarity detection and show that our model improves performance over strong neural baselines across a variety of English language datasets.We find that the addition of topics to BERT helps particularly with resolving domain-specific cases.

Existing datasets for scoring text pairs in terms of semantic similarity contain instances whose resolution differs according to the degree of difficulty.This paper proposes to distinguish obvious from non-obvious text pairs based on superficial lexical overlap and ground-truth labels.We characterise existing datasets in terms of containing difficult cases and find that recently proposed models struggle to capture the non-obvious cases of semantic similarity.We describe metrics that emphasise cases of similarity which require more complex inference and propose that these are used for evaluating systems for semantic similarity.