To understand scaling laws characterizing the loss landscape of large language models as a function of training data and parameter size, we need simple models relating properties of texts to their probabilities. Context-free grammars are a broad class of systems generating texts with long-range correlations, and which compactly encode the syntactic structure of human and computer languages. An ensemble of context-free grammars was proposed and dubbed the Random Language Model. It was shown that by varying the natural temperature-like parameter of the model one can encounter a transition between a simple ’babbling’ regime and a regime in which nontrivial information is carried. Analytical results using a field-theoretic method have been limited to the babbling regime and the transition onset, and the existence of a true phase transition has been questioned. Here we show that the ‘energetic’ aspect of the model is governed by a Random Energy Model, where the thermodynamic limit of the latter becomes a scaling limit of the language ensemble, in which the number of hidden symbols diverges while the temperature vanishes, in a controlled manner. As a consequence, the language ensemble has a spectrum of singular temperatures and a condensation transition at a particular reduced temperature, in the scaling limit. New theory significantly improves the prediction of Shannon entropy and energy, and may rationalize the dependence of entropy rate on context length observed in LLMs.

The talk will not assume prior background in linguistics. Afterwards, Eric is happy to chat aboutnoise-control duality and Doi-Peliti.

If you wish to to attend this seminar online, please send an email to m.t.pham@uva.nl to receive the zoom-link.