Random Energies and Random Languages
DIEP seminar by Eric De Giuli
Eric De Giuli, Associate Professor at Toronto Metropolitan University, who is currently on sabbatical in Paris at Bouchaud’s research lab, will be giving a talk at the DIEP seminar about large language models and context-free grammars.
Event details from: Random Energies and Random Languages
- Date
- 8 October 2026
- Time
- 11:00 – 12:00
- Room
- Second-floor library
To understand scaling laws characterizing the loss landscape of large language models as a function of training data and parameter size, we need simple models relating properties of texts to their probabilities. Context-free grammars are a broad class of systems generating texts with long-range correlations, and which compactly encode the syntactic structure of human and computer languages. An ensemble of context-free grammars was proposed and dubbed the Random Language Model. It was shown that by varying the natural temperature-like parameter of the model one can encounter a transition between a simple ’babbling’ regime and a regime in which nontrivial information is carried. Analytical results using a field-theoretic method have been limited to the babbling regime and the transition onset, and the existence of a true phase transition has been questioned. Here we show that the ‘energetic’ aspect of the model is governed by a Random Energy Model, where the thermodynamic limit of the latter becomes a scaling limit of the language ensemble, in which the number of hidden symbols diverges while the temperature vanishes, in a controlled manner. As a consequence, the language ensemble has a spectrum of singular temperatures and a condensation transition at a particular reduced temperature, in the scaling limit. New theory significantly improves the prediction of Shannon entropy and energy, and may rationalize the dependence of entropy rate on context length observed in LLMs.
The talk will not assume prior background in linguistics. Afterwards, Eric is happy to chat aboutnoise-control duality and Doi-Peliti.
If you wish to to attend this seminar online, please send an email to m.t.pham@uva.nl to receive the zoom-link.