Neural networks offer good approximation to many tasks but consistently fail to reach perfect generalization, even when theoretical work shows that such perfect solutions can be expressed by certain architectures.Using the task of formal language learning, we focus on one simple formal language and show that the theoretically correct solution is in fact not an optimum of commonly used objectiveseven with regularization techniques that according to common wisdom should lead to simple weights and good generalization (L1, L2) or other meta-heuristics (early-stopping, dropout).On the other hand, replacing standard targets with the Minimum Description Length objective (MDL) results in the correct solution being an optimum.

Recent progress in Spoken Language Modeling has shown that learning language directly from speech is feasible.Generating speech through a pipeline that operates at the text level typically loses nuances, intonations, and nonverbal vocalizations.Modeling directly from speech opens up the path to more natural and expressive systems.On the other hand, speechonly systems require up to three orders of magnitude more data to catch up to their text-based counterparts in terms of their semantic abilities.We show that fine-tuning speech representation models on phoneme classification leads to more context-invariant representations, and language models trained on these units achieve comparable lexical comprehension to ones trained on hundred times more data.

Originally formalized with symbolic representations, syntactic trees may also be effectively represented in the activations of large language models (LLMs). Indeed, a ''Structural Probe'' can find a subspace of neural activations, where syntactically-related words are relatively close to one-another. However, this syntactic code remains incomplete: the distance between the Structural Probe word embeddings can represent the \emph{existence} but not the type and direction of syntactic relations. Here, we hypothesize that syntactic relations are, in fact, coded by the relative direction between nearby embeddings. To test this hypothesis, we introduce a ''Polar Probe'' trained to read syntactic relations from both the distance and the direction between word embeddings. Our approach reveals three main findings. First, our Polar Probe successfully recovers the type and direction of syntactic relations, and substantially outperforms the Structural Probe by nearly two folds. Second, we confirm that this polar coordinate system exists in a low-dimensional subspace of the intermediate layers of many LLMs and becomes increasingly precise in the latest frontier models. Third, we demonstrate with a new benchmark that similar syntactic relations are coded similarly across the nested levels of syntactic trees. Overall, this work shows that LLMs spontaneously learn a geometry of neural activations that explicitly represents the main symbolic structures of linguistic theory.

Abstract We train neural networks to optimize a Minimum Description Length score, that is, to balance between the complexity of the network and its accuracy at a task. We show that networks optimizing this objective function master tasks involving memory challenges and go beyond context-free languages. These learners master languages such as anbn, anbncn, anb2n, anbmcn +m, and they perform addition. Moreover, they often do so with 100% accuracy. The networks are small, and their inner workings are transparent. We thus provide formal proofs that their perfect accuracy holds not only on a given test set, but for any input sequence. To our knowledge, no other connectionist model has been shown to capture the underlying grammars for these languages in full generality.

We propose a general framework to study language emergence through signaling games with neural agents.Using a continuous latent space, we are able to (i) train using backpropagation, (ii) show that discrete messages nonetheless naturally emerge.We explore whether categorical perception effects follow and show that the messages are not compositional.

Several definitions of logical consequence have been proposed in many-valued logic, which coincide in the two-valued case, but come apart as soon as three truth values come into play. Those definitions include so-called pure consequence, order-theoretic consequence and mixed consequence. In this article, we examine whether those definitions together carve out a natural class of consequence relations. We respond positively by identifying a small set of properties that we see instantiated in those various consequence relations, namely truth-relationality, value-monotonicity, validity-coherence and a constraint of bivalence-compliance, provably replaceable by a structural requisite of nontriviality. Our main result is that the class of consequence relations satisfying those properties coincides exactly with the class of mixed consequence relations and their intersections, including pure consequence relations and order-theoretic consequence. We provide an enumeration of the set of those relations in finite many-valued logics of two extreme kinds: those in which truth values are well-ordered and those in which values between 0 and 1 are incomparable.