Transformer Layers as Painters

2024/07/12 by Qi Sun, Sun, Qi, Marc Pickett +5 · 1 voice · 25 citations
Computer Science · Engineering · #Art #Business #Electrical engineering #Engineering #Materials science #Painting #Structural Engineering and Vibration Analysis #Transformer #Visual arts #Voltage #cs.CL

paper · pdf · doi:10.48550/arxiv.2407.09298

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/07/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite their nearly universal adoption for large language models, the internal workings of transformers are not well understood. We aim to better understand the impact of removing or reorganizing information throughout the layers of a pretrained transformer. Such an understanding could both yield better usage of existing models as well as to make architectural improvements to produce new variants. We present a series of empirical studies on frozen models that show that the lower and final layers of pretrained transformers differ from middle layers, but that middle layers have a surprising amount of uniformity. We further show that some classes of problems have robustness to skipping layers, running the layers in an order different from how they were trained, or running the layers in parallel. Our observations suggest that even frozen pretrained models may gracefully trade accuracy for latency by skipping layers or running layers in parallel.

Cited by

• Loop the Loopies!
• A Brain-like Synergistic Core in LLMs Drives Behaviour and Learning
• Resting Neurons, Active Insights: Improving Input Sparsification for Large Language Models
• Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis
• Towards Understanding Layer Contributions in Tabular In-Context Learning Models
• Direct Multi-Token Decoding
• The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities
• Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
• A Review of Developmental Interpretability in Large Language Models
• Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignment
• Auto-Compressing Networks
• Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
• Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training
• Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs
• Zero-Shot Vision Encoder Grafting via LLM Surrogates
• LoKI: Low-damage Knowledge Implanting of Large Language Models
• RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding
• Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models
• MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
• BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models
• Do Language Models Use Their Depth Efficiently?
• LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
• Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision
• Identifying and Evaluating Inactive Heads in Pretrained LLMs
• Decoding Vision Transformers: the Diffusion Steering Lens

Discussions

• Transformer Layers as Painters [hn, 99 points, 10 comments]

Related

• Reservoir Transformers
• Improving Transformer Models by Reordering their Sublayers
• Simplifying Transformer Blocks
• Diversity of Transformer Layers: One Aspect of Parameter Scaling Laws
• Rethinking the Value of Transformer Components