We explore the idea of using the pre-trained BERT as a source of factual knowledge, analyze which components of the model are responsible for its ability to answer questions requiring factual knowledge, and study the transferability of the knowledge to downstream tasks. Our experiments show that the Language Modeling Head is indispensable for predicting facts, implying that transferability of any knowledge captured in the model is limited. While the dominant approach to researching how knowledge is stored in language models focuses on tailoring question formulation to optimize the retrieval quality, we find question patterns easily understood by humans that confuse BERT to the point that the answer does not make sense. The nature of the found patterns implies that the stored knowledge is fragile and based on token co-occurrence in the training set used during pre-training, rather than generalization or inference. Moreover, using a novel, hand-crafted dataset we show that BERT is vulnerable to common misconceptions, which could have fatal effects on downstream applications. Overall, we conclude that BERT offers low and unreliable performance out of the box. Jupyter notebooks with experiments are available on GitHub.11Project URL: https://github.com/tenpercent/knowledge-and-confusion

The success of language models based on the Transformer architecture appears to be inconsistent with observed anisotropic properties of representations learned by such models.We resolve this by showing, contrary to previous studies, that the representations do not occupy a narrow cone, but rather drift in common directions.At any training step, all of the embeddings except for the ground-truth target embedding are updated with gradient in the same direction.Compounded over the training set, the embeddings drift and share common components, manifested in their shape in all the models we have empirically tested.Our experiments show that isotropy can be restored using a simple transformation. 1

Natural language processing has improved substantially in the last few years due to the increased computational power and availability of text data. Bidirectional Encoder Representations from Transformers (BERT) have further improved the performance by using an auto-encoding model that incorporates larger bidirectional contexts. However, the underlying mechanisms of BERT for its effectiveness are not well understood. In the paper we investigate how the BERT architecture and its pretraining protocol affect the geometry of its embeddings and the effectiveness of its features for classification tasks. As an autoencoding model, during pre-training, it produces representations that are context dependent and at the same time must be able to "reconstruct" the original input sentences. The complex interactions of the two via transformers lead to interesting geometric properties of the embeddings and subsequently affect the inherent discriminability of the resulting representations. Our experimental results illustrate that the BERT models do not produce "effective" contextualized representations for words and their improved performance may mainly be due to fine-tuning or classifiers that model the dependencies explicitly by encoding syntactic patterns in the training data.

Anecdotal evidence suggests the Variety of Big data is one of the most challenging problems in Computer Science research today [1]. First, Big data arrives from a myriad of data sources, hence its shape and flavor differ. Second, hundreds of different Big data management systems support different APIs, storage/indexing schemes, and expose data to the users through their data model lens, each specific to their own system. All of these offer a significant impediment for Big data users who just want an easy to use interface to all relevant data regardless of its shape, format, size, and a back-end system used to store it. Naturally, these differences also complicate development of any analytical algorithms on top of large-scale, heterogeneous datasets. Here we describe HYBRID.POLY- a consolidated in-memory polystore engine [2], designed to support heterogeneous large-scale data and interactively process complex analytical work-loads. We execute and evaluate several popular analytical work-loads including Data Fusion, Machine Learning, and Music search at scale.

Anecdotal evidence suggests that Variety is one of the most challenging problems in Big data research [1]. Different data providers use different data models and formats to represent their data, which causes significant impediment to data scientists, whose goal is to make sense of all relevant data regardless of the source. Hybrid.Poly [2], [3] is the analytical polystore data management system designed to make all data accessible to the analyst, oblivious of the source differences.In this paper, we focus on the in-depth analysis and performance evaluation of the Linear Algebra extensions added to the Hybrid.Poly language.

Hybrid.poly is an in-memory polystore data management system, able to ingest various kinds of data and run complex analytical workloads on the ingested data [Gubanov, 2017], [Podkorytov et al., 2017]. Hybrid.JSON, a part of [Gubanov, 2017] focuses on ingesting and querying JSON documents in the polystore.