This paper presents an information-theoretical model of syntactic generalization.We study syntactic generalization from the perspective of the capacity to disentangle semantic and structural information, emulating the human capacity to assign a grammaticality judgment to semantically nonsensical sentences.In order to isolate the structure, we propose to represent the probability distribution behind a corpus as the product of the probability of a semantic context and the probability of a structure, the latter being independent of the former.We further elaborate the notion of abstraction as a relaxation of the property of independence.It is based on the measure of structural and contextual information for a given representation.We test abstraction as an optimization objective on the task of inducing syntactic categories from natural language data and show that it significantly outperforms alternative methods.Furthermore, we find that when syntax-unaware optimization objectives succeed in the task, their success is mainly due to an implicit disentanglement process rather than to the model structure.On the other hand, syntactic categories can be deduced in a principled way from the independence between structure and context.

This paper is a theoretical contribution to the debate on the learnability of syntax from a corpus without explicit syntax-specific guidance.Our approach originates in the observable structure of a corpus, which we use to define and isolate grammaticality (syntactic information) and meaning/pragmatics information.We describe the formal characteristics of an autonomous syntax and show that it becomes possible to search for syntax-based lexical categories with a simple optimization process, without any prior hypothesis on the form of the model.

Word embeddings are used with success for a variety of tasks involving lexical semantic similarities between individual words.Using unsupervised methods and just cosine similarity, encouraging results were obtained for analogical similarities.In this paper, we explore the potential of pre-trained word embeddings to identify generic types of semantic relations in an unsupervised experiment.We propose a new relational similarity measure based on the combination of word2vec's CBOW input and output vectors which outperforms alternative vector representations, when used for unsupervised clustering on SemEval 2010 Relation Classification data.