TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios

2025/05/19 by Wei, Shaohang, Wei Li, Li, Wei +12 · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Constraint Satisfaction and Optimization #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.12891

openalex publication_date 2025/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Temporal reasoning is pivotal for Large Language Models (LLMs) to comprehend the real world. However, existing works neglect the real-world challenges for temporal reasoning: (1) intensive temporal information, (2) fast-changing event dynamics, and (3) complex temporal dependencies in social interactions. To bridge this gap, we propose a multi-level benchmark TIME, designed for temporal reasoning in real-world scenarios. TIME consists of 38,522 QA pairs, covering 3 levels with 11 fine-grained sub-tasks. This benchmark encompasses 3 sub-datasets reflecting different real-world challenges: TIME-Wiki, TIME-News, and TIME-Dial. We conduct extensive experiments on reasoning models and non-reasoning models. And we conducted an in-depth analysis of temporal reasoning performance across diverse real-world scenarios and tasks, and summarized the impact of test-time scaling on temporal reasoning capabilities. Additionally, we release TIME-Lite, a human-annotated subset to foster future research and standardized evaluation in temporal reasoning. The code is available at https://github.com/sylvain-wei/TIME , the dataset is available at https://huggingface.co/datasets/SylvainWei/TIME , and the project page link is https://sylvain-wei.github.io/TIME/ .

Citations

• FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
• REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
• TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues
• Qwen2.5-1M Technical Report
• ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events
• Qwen2.5 Technical Report
• Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?
• Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Evaluating Very Long-Term Conversational Memory of LLM Agents
• Large Language Models Can Learn Temporal Reasoning
• SCTc-TE: A Comprehensive Formulation and Benchmark for Temporal Event Forecasting
• TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models
• Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning
• Once Upon a Time in Graph: Relative-Time Pretraining for Complex Temporal Reasoning
• Time-Aware Representation Learning for Time-Sensitive Question Answering
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models
• FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
• SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning
• Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models
• RealTime QA: What's the Answer Right Now?
• StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
• Training Verifiers to Solve Math Word Problems
• SituatedQA: Incorporating Extra-Linguistic Contexts into QA
• A Dataset for Answering Time-Sensitive Questions
• Time-Aware Language Models as Temporal Knowledge Bases
• TIMEDIAL: Temporal Commonsense Reasoning in Dialog
• Measuring Mathematical Problem Solving With the MATH Dataset
• ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning
• Temporal Reasoning on Implicit Events from Distant Supervision
• STARC: Structured Annotations for Reading Comprehension
• "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding
• Fine-Grained Temporal Relation Extraction

Cited by

• Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents
• CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrency
• If I Could Turn Back Time: Temporal Reframing as a Historical Reasoning Task for LLMs
• TiMoE: Time-Aware Mixture of Language Experts
• HauntAttack: When Attack Follows Reasoning as a Shadow
• Verifier-Induced Support Reshaping in On-Policy Optimization
• When Silence Is Golden: Can LLMs Learn to Abstain in Temporal QA and Beyond?

Related

• TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models
• TRAM: Benchmarking Temporal Reasoning for Large Language Models
• Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
• Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Models
• Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models