BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning

2025/08/29 by João Guilherme Alves Santos, Santos, João Guilherme Alves, Giovana Kerche Bonás +3 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2508.21294

openalex publication_date 2025/08/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the growing capabilities of Large Language Models (LLMs), there is an increasing need for robust evaluation methods, especially in multilingual and non-English contexts. We present an updated version of the BLUEX dataset, now including 2024-2025 exams and automatically generated image captions using state-of-the-art models, enhancing its relevance for data contamination studies in LLM pretraining. Captioning strategies increase accessibility to text-only models by more than 40%, producing 1,422 usable questions, more than doubling the number in the original BLUEX. We evaluated commercial and open-source LLMs and their ability to leverage visual context through captions.

Citations

• TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Time
• Qwen2.5 Technical Report
• GPT-4o System Card
• The Llama 3 Herd of Models
• SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
• A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods
• MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models
• Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• BooookScore: A systematic exploration of book-length summarization in the era of LLMs
• SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
• Using an LLM to Help With Code Understanding
• BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams
• Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion
• Perception Test: A Diagnostic Benchmark for Multimodal Video Models
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
• TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
• Planning with Large Language Models for Code Generation
• Benchmarking Large Language Models for News Summarization
• Internet-augmented language models through few-shot prompting for open-domain question answering
• Training language models to follow instructions with human feedback
• Evaluating Large Language Models Trained on Code
• Measuring Massive Multitask Language Understanding
• Language Models as Knowledge Bases?

Cited by

• Curió-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining

Related

• CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
• CapsFusion: Rethinking Image-Text Data at Scale
• Multi-LLM Collaborative Caption Generation in Scientific Documents
• Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize\n Long-Tail Visual Concepts