TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention

2026/08/03 by Avni Mittal, Avinash Anand, Ashutosh Kumar +7
Computer Science · #cs.CL #cs.AI #cs.LG

paper · pdf

arxiv created 2026/08/03 · arxiv updated 2026/08/04

Abstract

Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour? We define TextNCA, a 1D causal windowed-attention realisation of the Neural Cellular Automaton primitive, and study a hierarchical variant that cascades three stages with windows w ∈ \8, 32, 128\ and T_s shared-weight iterations per stage, all on WikiText-103 at roughly 30M parameters and 60k training steps. The model does not match a parameter-matched Transformer at this scale (Hier-TextNCA 60.3 vs. Transformer-6L 52.8 and Transformer-12L 44.7 PPL), so we treat it as an analytical probe rather than a proposed alternative. The behaviour we observe is largely explained by the staged narrow-to-wide schedule: a non-iterating sliding-window Transformer that reuses the same schedule comes within +4.1 PPL of the iterated model, while reversing, flattening, or breaking the monotonic ordering of the schedule costs between +16.7 and +70.8 PPL. Iteration adds a smaller bounded benefit on top of the schedule, with a clear optimum at T_s=4 and a U-shaped degradation beyond it. The GRU gate and learned per-step embeddings are required for that benefit to appear, and training with random T_s yields an inference-time iteration-count knob at the cost of substantially higher absolute PPL. We position the work as a controlled reading of which parts of NCA-style computation carry the weight in language modelling.

Citations

• Training Language Models via Neural Cellular Automata
• Reasoning with Latent Thoughts: On the Power of Looped Transformers
• Latent Iteration as Renormalization: Inference-Time Recurrence in a Depth-Recurrent Transformer Flows Attention Geometry Toward the SYK Conformal Fixed Point
• Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA
• Gated Linear Attention Transformers with Hardware-Efficient Training
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Hyena Hierarchy: Towards Larger Convolutional Language Models
• Med-NCA: Robust and Lightweight Segmentation with Neural Cellular Automata
• Texture Generation with Neural Cellular Automata
• Longformer: The Long-Document Transformer
• Scaling Laws for Neural Language Models
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Deep Equilibrium Models
• Similarity of Neural Network Representations Revisited
• Universal Transformers
• Pointer Sentinel Mixture Models
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Training Deep Nets with Sublinear Memory Cost
• Adaptive Computation Time for Recurrent Neural Networks
• Neural GPUs Learn Algorithms