Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information

2025/02/20 by Yein Park, Chanwoong Yoon, Park, Yein +7 · 2 voices · 5 citations
Computer Science · Psychology · Social Sciences · #Artificial intelligence #Cognitive psychology #Computer science #Language and cultural evolution #Natural Language Processing Techniques #Psychology #Recall #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2502.14258

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

While the ability of language models to elicit facts has been widely investigated, how they handle temporally changing facts remains underexplored. We discover Temporal Heads, specific attention heads that primarily handle temporal knowledge, through circuit analysis. We confirm that these heads are present across multiple models, though their specific locations may vary, and their responses differ depending on the type of knowledge and its corresponding years. Disabling these heads degrades the model's ability to recall time-specific knowledge while maintaining its general capabilities without compromising time-invariant and question-answering performances. Moreover, the heads are activated not only numeric conditions ("In 2004") but also textual aliases ("In the year ..."), indicating that they encode a temporal dimension beyond simple numerical representation. Furthermore, we expand the potential of our findings by demonstrating how temporal knowledge can be edited by adjusting the values of these heads.

Citations

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Time is Encoded in the Weights of Finetuned Language Models

Cited by

• Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity
• ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attack
• Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training
• Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research
• Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering

Discussions

• Ablation studies demonstrate that language models store time-dependent information in special "temporal heads." E.g. "In YYYY, the President of South Korea was ..." #MLSky arxiv.org/abs/2502.14258 [bsky, 41 points, 2 comments]
• PAPER: There are specialised attention heads in LLMs that specifically process time-dependent knowledge. These "temporal heads" can manipulated temporally precise facts - without affecting the rest of [bsky, 1 points, 1 comments]

Related

• Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time
• Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models
• Time-Aware Language Models as Temporal Knowledge Bases
• ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains
• Discrete Minds in a Continuous World: Do Language Models Know Time Passes?