Layer-wise Positional Bias in Short-Context Language Modeling

2026/01/31 by Maryam Rahimi, Mahdi Nouri, Yadollah Yaghoobzadeh
Computer Science · #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2601.04098

arxiv created 2026/08/06 · arxiv updated 2026/08/07

Abstract

Transformer language models systematically prefer tokens at specific input positions regardless of semantic relevance---a phenomenon known as positional bias. Prior work characterizes this bias in model behavior through performance drops in long-context tasks or in model architecture through attention-based analyses. However, it remains unmeasured how input positions actually drive predictions layer by layer. We introduce a layer conductance framework within a sliding-window design, applied to short-context next-word prediction to isolate model-internal behavior from task and context-window pressure. The resulting layer-wise positional importance profiles are stable across diverse texts and lexical scrambling, confirming they reflect model-internal structure. Characterizing how these profiles evolve across depth, we find recency bias increases monotonically while primacy bias is subtle and diminishes. We also find that this positional bias is not uniform across word types: function words exhibit higher recency bias while content words show higher primacy bias.

Citations

• Fine-grained Analysis of Brain-LLM Alignment through Input Attribution
• Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs
• Positional Biases Shift as Inputs Approach Context Window Limits
• Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs
• Do RAG Systems Really Suffer From Positional Bias?
• Explanations of Large Language Models Explain Language Representations in the Brain
• On the Emergence of Position Bias in Transformers
• Unraveling Token Prediction Refinement and Identifying Essential Layers in Language Models
• The Llama 3 Herd of Models
• Eliminating Position Bias of Language Models: A Mechanistic Approach
• Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
• Efficient Streaming Language Models with Attention Sinks
• Decoding Layer Saliency in Language Transformers
• Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias
• Lost in the Middle: How Language Models Use Long Contexts
• Large Language Models Can Be Easily Distracted by Irrelevant Context
• Measuring the Mixing of Contextual Information in the Transformer
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• Captum: A unified and generic model interpretability library for PyTorch
• Quantifying Attention Flow in Transformers
• Look at the First Sentence: Position Bias in Question Answering
• Telling BERT's full story: from Local Attention to Global Aggregation
• Is Attention Interpretable?
• BERT Rediscovers the Classical NLP Pipeline
• Attention is not Explanation
• Computationally Efficient Measures of Internal Neuron Importance
• How Important Is a Neuron?
• Attention Is All You Need