2026/05/07 by Philipp Stecher, Sandro Radovanović, Vlasta Sikimić +2 · 1 voice
Biochemistry, Genetics and Molecular Biology · Computer Science · Neuroscience · Psychology · Social Sciences · #Feature (linguistics) #Joint (building) #Language Development and Disorders #Language and cultural evolution #Neurobiology of Language and Bilingualism #Scaffold #Scaffold protein #Selection (genetic algorithm) #cs.AI #q-bio.NC
paper · pdf · doi:10.48550/arxiv.2605.18789
openalex publication_date 2026/05/07 · arxiv published 2026/05/07 · arxiv updated 2026/05/07 · openalex created_date 2026/05/21 · openalex updated_date 2026/07/28
Features in language models have life history: they emerge, persist, and die during training, yet the importance of that history remains largely unexplored. We find evidence of a persistent representational backbone, which we identify in Pythia-160M and -410M as the carrier scaffold: ∼50 sparse features with stable life histories, around which the model's representational structure organises. It has four properties. (i)~It assembles early: features emerge, die, and reorganise ∼40 × faster in the first 1% of training than afterwards, and the scaffold is already largely fixed by then. (ii)~It is load-bearing: joint cross-layer ablation identifies the carriers as far more load-bearing than any count-matched non-scaffold population, a gap invisible to per-firing single-feature methods. (iii)~Function precedes direction: which features will become carriers is already predictable from training-onset firing patterns alone, correctly distinguishing future carriers from non-carriers in 4 of 5 cases, before the geometry has settled. (iv)~It seeds subsequent development: by the end of training, scaffold carriers have recruited 64% of all active features into the scaffold hierarchy. Life history is consistent with a two-phase account of training: selection appears to largely determine the scaffold in the first 1%; the remaining 99% appears to calibrate geometry around a substrate already set.