Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs

2025/02/07 by Rohit Saxena, Aryo Pradipta Gema, Saxena, Rohit +3 · 7 voices · 4 citations
Computer Science · #Advanced Computational Techniques and Applications #Semantic Web and Ontologies #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2502.05092

Abstract

Understanding time from visual representations is a fundamental cognitive skill, yet it remains a challenge for multimodal large language models (MLLMs). In this work, we investigate the capabilities of MLLMs in interpreting time and date through analogue clocks and yearly calendars. To facilitate this, we curated a structured dataset comprising two subsets: 1) ClockQA, which comprises various types of clock styles-standard, black-dial, no-second-hand, Roman numeral, and arrow-hand clocks-paired with time related questions; and 2) CalendarQA, which consists of yearly calendar images with questions ranging from commonly known dates (e.g., Christmas, New Year's Day) to computationally derived ones (e.g., the 100th or 153rd day of the year). We aim to analyse how MLLMs can perform visual recognition, numerical reasoning, and temporal inference when presented with time-related visual data. Our evaluations show that despite recent advancements, reliably understanding time remains a significant challenge for MLLMs.

Cited by

• SPAN: Benchmarking and Improving Cross-Calendar Temporal Reasoning of Large Language Models
• Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench
• Fitting the Message to the Moment: Designing Calendar-Aware Stress Messaging with Large Language Models
• Have Multimodal Large Language Models (MLLMs) Really Learned to Tell the Time on Analog Clocks?

Discussions

• Why is it so difficult for Large Language Models to properly read time? arxiv.org/abs/2502.05092 [bsky, 15 points, 0 comments]
• LLMs can tackle math olympiad problems, but… can they read a clock or calendar? 🕰️📆 Our experiments reveal surprising failures in temporal reasoning—multimodal models struggle with analogue clock re [bsky, 13 points, 0 comments]
• Paper: arxiv.org/abs/2502.05092 [bsky, 8 points, 0 comments]
• Clock and Calendar Understanding Challenges in Multimodal LLMs [hn, 1 points, 0 comments]
• Updated Preprint: "Lost in #Time: Clock and Calendar Understanding Challenges in Multimodal #LLMs" (via #arXiv) arxiv.org/abs/2502.05092 #ai [bsky, 0 points, 0 comments]
• Peut-on regarder pourquoi les LLM ne pouvaient il y a deux mois pas lire les montres à aiguilles et l'heure analogique ? arxiv.org/abs/2502.05092 [bsky, 0 points, 0 comments]
• Great! It is always valuable to have benchmarks on AI “breakthroughs”. I believe that LLM's can well be trained to improve this particular capability. arxiv.org/abs/2502.05092 [bsky, 0 points, 0 comments]

Related

• Have Multimodal Large Language Models (MLLMs) Really Learned to Tell the Time on Analog Clocks?
• SPAN: Benchmarking and Improving Cross-Calendar Temporal Reasoning of Large Language Models
• Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki
• Date Fragments: A Hidden Bottleneck of Tokenization for Temporal Reasoning
• ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events