Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training

2025/09/08 by Andrei Baroian, Baroian, Andrei, Kasper Notebomer +1 · 1 citation
Engineering · Medicine · #Artificial Intelligence (cs.AI) #Cardiac Valve Diseases and Treatments #Computation and Language (cs.CL) #FOS: Computer and information sciences #Orthopaedic implants and arthroplasty #Tunneling and Rock Mechanics

paper · pdf · doi:10.48550/arxiv.2509.06518

openalex publication_date 2025/09/08 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Transformer-based language models traditionally use uniform (isotropic) layer sizes, yet they ignore the diverse functional roles that different depths can play and their computational capacity needs. Building on Layer-Wise Scaling (LWS) and pruning literature, we introduce three new LWS variants - Framed, Reverse, and Crown - that redistribute FFN widths and attention heads via two or three-point linear interpolation in the pre-training stage. We present the first systematic ablation of LWS and its variants, on a fixed budget of 180M parameters, trained on 5B tokens. All models converge to similar losses and achieve better performance compared to an equal-cost isotropic baseline, without a substantial decrease in training throughput. This work represents an initial step into the design space of layer-wise architectures for pre-training, but future work should scale experiments to orders of magnitude more tokens and parameters to fully assess their potential.

Citations

• LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions
• Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective
• Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training
• Layer by Layer: Uncovering Hidden Representations in Language Models
• DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs
• Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
• OpenELM: An Efficient Language Model Family with Open Training and Inference Framework
• Model Compression and Efficient Inference for Large Language Models: A Survey
• Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layers
• The Impact of Depth on Compositional Generalization in Transformer Language Models
• A Survey on Model Compression for Large Language Models
• Training Compute-Optimal Large Language Models
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Fast Transformer Decoding: One Write-Head is All You Need
• Crowdsourcing Multiple Choice Science Questions

Cited by

• Tapered Language Models

Related

• LESA: Learnable LLM Layer Scaling-Up
• Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers
• The Curse of Depth in Large Language Models
• Benchmarking down-scaled (not so large) pre-trained language models
• DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs