MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

2024/06/03 by Yubo Wang, Wang, Yubo, Xueguang Ma +31 · 786 citations
Computer Science · #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2406.01574

Abstract

In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) have been pivotal in pushing the boundaries of what AI can achieve in language comprehension and reasoning across diverse domains. However, as models continue to improve, their performance on these benchmarks has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU. Our experimental results show that MMLU-Pro not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, which is in stark contrast to the findings on the original MMLU, indicating that MMLU-Pro includes more complex reasoning questions. Our assessments confirm that MMLU-Pro is a more discriminative benchmark to better track progress in the field.

Cited by

• Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
• Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
• Reinforcement Learning via Self-Distillation
• Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
• Diversity or Precision? A Deep Dive into Next Token Prediction
• HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery
• When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-offs
• LiveProteinBench: A Contamination-Free Benchmark for Assessing Models' Specialized Capabilities in Protein Science
• HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference
• DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
• Attention Residuals
• Explainable Model Routing for Agentic Workflows
• ScholarSearch: Benchmarking Scholar Searching Ability of LLMs
• Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models
• Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
• CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
• LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation
• What Does Chain-of-Thought Contribute at Probe Time? Evidence for Local Co-Occurrence Activation
• Do LLMs Experience an Internal Polylogue? Investigating Reasoning through the Lens of Personas
• Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact
• QMBench: A Research Level Benchmark for Quantum Materials Research
• Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM
• Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model
• LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamics
• Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuits
• Adversarial Robustness of Vision in Open Foundation Models
• Xiaomi MiMo-VL-Miloco Technical Report
• Trust-Region Adaptive Policy Optimization
• Refusal Steering: Fine-grained Control over LLM Refusal Behaviour for Sensitive Topics
• AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding
• Sigma-MoE-Tiny Technical Report
• T5Gemma 2: Seeing, Reading, and Understanding Longer
• Conditional Coverage Diagnostics for Conformal Prediction
• Grammar Search for Multi-Agent Systems
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value
• ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
• MedCEG: Reinforcing Verifiable Medical Reasoning with Critical Evidence Graph
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
• SIGMA: An AI-Empowered Training Stack on Early-Life Hardware
• Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
• Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
• LLMs Can Assist with Proposal Selection at Large User Facilities
• AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding
• FlipLLM: Efficient Bit-Flip Attacks on Multimodal LLMs using Reinforcement Learning
• RIFT: A Scalable Methodology for LLM Accelerator Fault Assessment using Reinforcement Learning
• MOA: Multi-Objective Alignment for Role-Playing Agents
• LLaDA2.0: Scaling Up Diffusion Language Models to 100B
• Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
• rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection
• Towards a Science of Scaling Agent Systems
• Single-Agent Scaling Fails Multi-Agent Intelligence: Towards Foundation Models with Native Multi-Agent Intelligence
• Training Language Models to Use Prolog as a Tool
• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
• Improved Supervised Fine-Tuning for Large Language Models to Mitigate Catastrophic Forgetting
• JT-DA: Enhancing Data Analysis with Tool-Integrated Table Reasoning Large Language Models
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
• A Latent Variable Framework for Scaling Laws in Large Language Models
• Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy
• Evolutionary System 2 Reasoning: An Empirical Proof
• Sell Data to AI Algorithms Without Revealing It: Secure Data Valuation and Sharing via Homomorphic Encryption
• Qwen3.5-Omni Technical Report
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• ADAPT: Learning Task Mixtures for Budget-Constrained Instruction Tuning
• Jina-VLM: Small Multilingual Vision Language Model
• PretrainZero: Reinforcement Active Pretraining
• MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages
• Guided Self-Evolving LLMs with Minimal Human Supervision
• E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
• DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Models
• InstructLR: A Scalable Approach to Create Instruction Dataset for Under-Resourced Languages
• Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction
• OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning
• Revisiting Generalization Across Difficulty Levels: It's Not So Easy
• OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency
• Unexplored flaws in multiple-choice VQA evaluations
• On the Limits of Innate Planning in Large Language Models
• From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
• ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning
• Qwen3-VL Technical Report
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
• Structured Prompting Enables More Robust Evaluation of Language Models
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems
• Majority of the Bests: Improving Best-of-N via Bootstrapping
• A²Flow: Automating Agentic Workflow Generation via Self-Adaptive Abstraction Operators
• Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT
• E³-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
• Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
• FlipVQA: Scaling Multi-modal Instruction Tuning via Textbook-to-Knowledge Synthesis
• Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
• EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
• Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
• Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distribution
• Stealth Fine-Tuning: Efficiently Breaking Alignment in RVLMs Using Self-Generated CoT
• Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
• Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
• Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
• Virtual Width Networks
• When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
• ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference
• Reasoning about Intent for Ambiguous Requests
• Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis for Large Reasoning Models
• AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment
• Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models
• ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction
• AlphaResearch: Accelerating New Algorithm Discovery with Language Models
• SPEAR-MM: Selective Parameter Evaluation and Restoration via Model Merging for Efficient Financial LLM Adaptation
• SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning
• DynaAct: Large Language Model Reasoning with Dynamic Action Spaces
• SERL: Self-Examining Reinforcement Learning on Open-Domain
• Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
• Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering
• RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Services
• Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
• S-DAG: A Subject-Based Directed Acyclic Graph for Multi-Agent Heterogeneous Reasoning
• You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
• What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Models
• LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation
• LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs
• Can a Small Model Learn to Look Before It Leaps? Dynamic Learning and Proactive Correction for Hallucination Detection
• Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges
• Motif 2 12.7B technical report
• Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale
• Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm
• NVIDIA Nemotron Nano V2 VL
• ChiMDQA: Towards Comprehensive Chinese Document QA with Fine-grained Evaluation
• From Prompts to Power: Measuring the Energy Footprint of LLM Inference
• Zero-shot data citation function classification using transformer-based large language models (LLMs)
• Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
• POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarios
• CGES: Confidence-Guided Early Stopping for Efficient and Accurate Self-Consistency
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
• LongCat-Flash-Omni Technical Report
• The End of Manual Decoding: Towards Truly End-to-End Language Models
• Think Outside the Policy: In-Context Steered Policy Optimization
• Lean4Physics: Comprehensive Reasoning Framework for College-level Physics in Lean4
• QuantumBench: A Benchmark for Quantum Problem Solving
• Gaperon: A Peppered English-French Generative Language Model Suite
• Adaptively Robust LLM Monitoring via Activation Watermarking
• Zero Reinforcement Learning Towards General Domains
• Metis: Memory Foundation Model
• When benchmark inferences do not compose: Projectibility in AI evaluation
• Will Scaling Improve Social Simulation with LLMs?
• The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliability
• Enhancing Multi-Agent Communication through Attention Steering with Context Relevance
• ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
• VerIF: Verification Engineering for Reinforcement Learning in Instruction Following
• Arcee Trinity Large Technical Report
• PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies
• HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
• RAVR: Reference-Answer-guided Variational Reasoning for Large Language Models
• Activation-Space Personality Steering: Hybrid Layer Selection for Stable Trait Control in LLMs
• AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis
• SPICE: Self-Play In Corpus Environments Improves Reasoning
• Dissecting Role Cognition in Medical LLMs via Neuronal Ablation
• MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling
• Parallel Loop Transformer for Efficient Test-Time Computation Scaling
• Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices
• APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training
• ChessQA: Evaluating Large Language Models for Chess Understanding
• Multi-Agent Evolve: LLM Self-Improve through Co-evolution
• VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation
• A Survey on LLM Mid-Training
• Knocking-Heads Attention
• Towards Scalable Oversight via Partitioned Human Supervision
• HypoSpace: A Diagnostic Benchmark for Set-Valued Hypothesis Generation under Underdetermination and Sublinear Coverage Bounds
• What Does It Take to Build a Performant Selective Classifier?
• Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values
• Why DPO is a Misspecified Estimator and How to Fix It
• DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference
• Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs
• Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model
• A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
• The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives
• Strengthening LLMs for Tabular Prediction with Structural Priors
• JT-Safe: Intrinsically Enhancing the Safety and Trustworthiness of LLMs
• Watermark Robustness and Radioactivity May Be at Odds in Federated Learning
• A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• Neuronal Group Communication for Efficient Neural representation
• KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models
• SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradients
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
• You May Speak Freely: Improving the Fine-Grained Visual Recognition Capabilities of Multimodal Large Language Models with Answer Extraction
• Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoning
• Selective Labeling with False Discovery Rate Control
• Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following
• MorphoBench: A Benchmark with Difficulty Adaptive to Model Reasoning
• RLSR: Reinforcement Learning with Supervised Reward Outperforms SFT in Instruction Following
• Toward Cybersecurity-Expert Small Language Models
• NOSA: Native and Offloadable Sparse Attention
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
• Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
• Selective Adversarial Attacks on LLM Benchmarks
• Schema for In-Context Learning
• Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems
• The Harder The Better: Maintaining Supervised Fine-tuning Generalization with Less but Harder Data
• Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models
• HoneyBee: Data Recipes for Vision-Language Reasoners
• A²FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning
• ExpVid: A Benchmark for Experiment Video Understanding & Reasoning
• AgentCaster: Reasoning-Guided Tornado Forecasting
• From to : Multidimensional Supervision of Reasoning Process for LLM Optimization
• Neural Weight Compression for Language Models
• ExGRPO: Learning to Reason from Experience
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration
• One Token Embedding Is Enough to Deadlock Your Large Reasoning Model
• MedCoAct: Confidence-Aware Multi-Agent Collaboration for Complete Clinical Decision
• Trace Length is a Simple Uncertainty Signal in Reasoning Models
• STEAM: A Semantic-Level Knowledge Editing Framework for Large Language Models
• SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
• The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities
• Learning Reasoning Rewards from Expert Demonstrations with Inverse Reinforcement Learning
• Brick: Spatial Capability Routing for the Mixture-of-Models (MoM) Paradigm
• Decoupled DiLoCo for Resilient Distributed Pre-training
• The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More
• Don't Throw Away Your Pretrained Model
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness
• ICL-Router: In-Context Learned Model Representations for LLM Routing
• FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
• Multimodal Policy Internalization for Conversational Agents
• KORMo: Korean Open Reasoning Model for Everyone
• Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL
• How Reliable is Language Model Micro-Benchmarking?
• Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
• ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping
• Lossless Vocabulary Reduction for Auto-Regressive Language Models
• When to Reason: Semantic Router for vLLM
• SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
• Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
• EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models
• Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
• Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
• SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora
• PeerRank: Autonomous LLM Evaluation Through Web-Grounded, Bias-Controlled Peer Review
• Mid-Training of Large Language Models: A Survey
• Efficient numeracy in language models through single-token number embeddings
• Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
• h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
• Grouped Differential Attention
• Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoning
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
• Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
• Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference
• InvThink: Premortem Reasoning for Safer Language Models
• Robustness assessment of large audio language models in multiple-choice evaluation
• Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
• MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
• Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
• The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
• Searching Meta Reasoning Skeleton to Guide LLM Reasoning
• Measuring Language Model Hallucinations Through Distributional Correctness
• MedReflect: Teaching Medical LLMs to Self-Improve via Reflective Correction
• COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier
• Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs
• Rethinking Reward Models for Multi-Domain Test-Time Scaling
• In-Place Feedback: Reliable Refinement for Multi-Turn Expert-LLM Collaboration
• Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
• Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
• A-VERT: Agnostic Verification with Embedding Ranking Targets
• Stochastic Self-Organization in Multi-Agent Systems
• Improving Metacognition and Uncertainty Communication in Language Models
• Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark
• AI Playing Business Games: Benchmarking Large Language Models on Managerial Decision-Making in Dynamic Simulations
• One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient
• Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities
• The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
• Nudging the Boundaries of LLM Reasoning
• Diversity-Incentivized Exploration for Versatile Reasoning
• RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMs
• UniAPL: A Unified Adversarial Preference Learning Framework for Instruct-Following
• The Era of Real-World Human Interaction: RL from User Conversations
• CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
• A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memory
• AdaThink-Med: Optimizing Inference-Time Compute for Medical Reasoning via Uncertainty Quantification
• SCI-Verifier: Scientific Verifier with Thinking
• ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scaling
• Beyond the Exploration-Exploitation Trade-off: A Hidden State Approach for LLM Reasoning in RLVR
• Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting
• CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems
• From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
• Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate
• d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• Variational Reasoning for Language Models
• When Does Reasoning Matter? A Controlled Study of Reasoning's Contribution to Model Performance
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
• COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
• No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
• SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification
• KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI
• Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
• Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond
• CLaw: Benchmarking Chinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis
• Predicting LLM Reasoning Performance with Small Proxy Model
• Look Before you Leap: Estimating LLM Benchmark Scores from Descriptions
• On Theoretical Interpretations of Concept-Based In-Context Learning
• Confidence-guided Refinement Reasoning for Zero-shot Question Answering
• Instruction Boundary: Quantifying Biases in LLM Reasoning under Various Coverage
• Thinking Augmented Pre-training
• TRUEBench: Can LLM Response Meet Real-world Constraints as Productivity Assistant?
• Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation
• What Does Your Benchmark Really Measure? A Framework for Robust Inference of AI Capabilities
• LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
• SKIMIX: Multi-Agent Harness-Time Scaling with Skill Mixture for Dynamic Harness Engineering
• Agent Harness Distillation: Inference-Time Harness Extraction and Exploitation in Autonomous Multi-Agent Systems
• Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
• Divergence Decoding: Training-Free Capability Fusion
• The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
• EMO: Pretraining Mixture of Experts for Emergent Modularity
• RELISH: LLM REgression with a Latent Iterative State Head
• Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities
• MentorCollab: Selective Large-to-Small Inference-Time Guidance for Efficient Reasoning
• The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation From Recognition to Reasoning
• Evaluating large language models for accuracy incentivizes hallucinations
• Proximal Supervised Fine-Tuning
• Reinforcement Learning on Pre-Training Data
• Introducing LongCat-Flash-Thinking: A Technical Report
• ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
• EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving
• Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Process
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
• Asking a Language Model for Diverse Responses
• Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs
• ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• RephQA: Evaluating Readability of Large Language Models in Public Health Question Answering
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression
• Reward Hacking Mitigation using Verifiable Composite Rewards
• GPO: Learning from Critical Steps to Improve LLM Reasoning
• Think, Verbalize, then Speak: Bridging Complex Thoughts and Comprehensible Speech
• CARGO: A Framework for Confidence-Aware Routing of Large Language Models
• Digging Into the Internal: Causality-Based Analysis of LLM Function Calling
• Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD
• Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement Learning
• Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
• GenExam: A Multidisciplinary Text-to-Image Exam
• Aegis: Automated Error Generation and Attribution for Multi-Agent Systems
• All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning
• LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactions
• Instance-level Randomization: Toward More Stable LLM Evaluations
• MALLM: Multi-Agent Large Language Models Framework
• MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables
• BenchECG and xECG: a benchmark and baseline for ECG foundation models
• Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning
• DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models
• Merge-of-Thought Distillation
• Bias after Prompting: Persistent Discrimination in Large Language Models
• WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding
• RL Fine-Tuning Heals OOD Forgetting in SFT
• MedBench-IT: A Comprehensive Benchmark for Evaluating Large Language Models on Italian Medical Entrance Examinations
• Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
• Hunyuan-MT Technical Report
• ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly
• Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
• Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving
• LongCat-Flash Technical Report
• Can Large Language Models Master Complex Card Games?
• ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics
• Balanced Actor Initialization: Stable RLHF Training of Distillation-Based Reasoning Models
• Standard vs. Modular Sampling: Best Practices for Reliable LLM Unlearning
• Can Multiple Responses from an LLM Reveal the Sources of Its Uncertainty?
• Intern-S1: A Scientific Multimodal Foundation Model
• TCIA: A Task-Centric Instruction Augmentation Method for Instruction Finetuning
• Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning
• Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis
• Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?
• Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
• Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• LaQual: A Novel Framework for Automated Evaluation of LLM App Quality
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Hermes 4 Technical Report
• Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
• LongRecall: A Structured Approach for Robust Recall Evaluation in Long-Form Text
• XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning
• Credence Calibration Game? Calibrating Large Language Models through Structured Play
• ZPD-SCA: Unveiling the Blind Spots of LLMs in Assessing Students' Cognitive Abilities
• Expertise-aware Multi-LLM Recruitment and Collaboration for Medical Decision-Making
• COCO: Cognitive Operating System with Continuous Oversight for Multi-Agent Workflow Reliability
• Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
• Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
• CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction
• DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoning
• Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
• The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution
• AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoning
• LARC: Towards Human-level Constrained Retrosynthesis Planning through an Agentic Framework
• STEM: Efficient Relative Capability Evaluation of LLMs through Structured Transition Samples
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• Flexible Personalized Split Federated Learning for On-Device Fine-Tuning of Foundation Models
• Efficient Forward-Only Data Valuation for Pretrained LLMs and VLMs
• Apriel-Nemotron-15B-Thinker
• Scaling Up Active Testing to Large Language Models
• IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization
• 3DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs
• TeamMedAgents: Pareto-Efficient Multi-Agent Medical Reasoning Through Teamwork Theory
• Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
• AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance
• Sample-efficient LLM Optimization with Reset Replay
• UR²: Unify RAG and Reasoning through Reinforcement Learning
• VISTAR:A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation
• Can Large Models Fool the Eye? A New Turing Test for Biological Animation
• SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges
• InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs to Enhance Reasoning Capabilities
• Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning
• TRAIL: Joint Inference and Refinement of Knowledge Graphs with Large Language Models
• From "Aha Moments" to Controllable Thinking: Toward Meta-Cognitive Reasoning in Large Reasoning Models via Decoupled Reasoning and Control
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward
• Thinking with Nothinking Calibration: A New In-Context Learning Paradigm in Reasoning Large Language Models
• CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment
• FAIR-Pruner: Leveraging Tolerance of Difference for Flexible Automatic Layer-Wise Neural Network Pruning
• Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following
• ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks
• Large-Scale Diverse Synthesis for Mid-Training
• LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points
• Towards Efficient Medical Reasoning with Minimal Fine-Tuning Data
• Prompting Science Report 3: I'll pay you or I'll kill you -- but will you care?
• MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language
• Multi-Layer Attention is the Amplifier of Demonstration Effectiveness
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models
• RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
• Cascaded Information Disclosure for Generalized Evaluation of Problem Solving Capabilities
• Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity
• ControlMed: Adding Reasoning Control to Medical Language Model
• Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
• Metric assessment protocol in the context of answer fluctuation on MCQ tasks
• ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge
• Can large language models assist choice modelling? Insights into prompting strategies and current models capabilities
• AgriEval: A Comprehensive Chinese Agricultural Benchmark for Large Language Models
• MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation
• Kimi K2: Open Agentic Intelligence
• Cultivating Helpful, Personalized, and Creative AI Tutors: A Framework for Pedagogical Alignment using Reinforcement Learning
• RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation
• Reasoning Models are Test Exploiters: Rethinking Multiple-Choice
• Flora: Effortless Context Construction to Arbitrary Length and Scale
• Adaptive Cluster Collaborativeness Boosts LLMs Medical Decision Support Capacity
• CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning
• Towards Effective Human-in-the-Loop Assistive AI Agents
• Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling
• Technical Report of TeleChat2, TeleChat2.5 and T1
• Are LLM Belief Updates Consistent with Bayes' Theorem?
• Byzantine-Robust Decentralized Coordination of LLM Agents
• Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling
• AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
• Mellum2 Technical Report
• stable-worldmodel-v1: Reproducible World Modeling Research and Evaluation
• ERNIE 5.0 Technical Report
• Towards Evaluting Fake Reasoning Bias in Language Models
• Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks
• MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
• Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
• Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models
• MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning
• A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning
• RePO: Replay-Enhanced Policy Optimization
• MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models
• GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture
• LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
• Med-REFL: Medical Reasoning Enhancement via Self-Corrected Fine-grained Reflection
• Learning to Reason Across Parallel Samples for LLM Reasoning
• AdaMuon: Adaptive Muon Optimizer
• EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes
• Step-wise Policy for Rare-tool Knowledge (SPaRK): Offline RL that Drives Diverse Tool Use in LLMs
• Open-Source LLMs Collaboration Beats Closed-Source LLMs: A Scalable Multi-Agent System
• HKGAI-V1: Towards Regional Sovereign Large Language Model for Hong Kong
• Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
• RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Services
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards
• Reinforce LLM Reasoning through Multi-Agent Reflection
• From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation
• A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench
• KAT-V1: Kwai-AutoThink Technical Report
• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
• RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrieval
• Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
• Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code
• TextCloak: Thwarting Unauthorized LLM Exploitation via RL-Driven Unlearnable Text
• Perception-Aware Policy Optimization for Multimodal Reasoning
• BlueLM-2.5-3B Technical Report
• Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning
• Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
• Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systems
• Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
• Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning
• Pre-Trained Policy Discriminators are General Reward Models
• CortexDebate: Debating Sparsely and Equally for Multi-Agent Debate
• Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
• BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
• Answer Matching Outperforms Multiple Choice for Language Model Evaluation
• Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks
• EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
• Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery
• Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Models
• Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
• NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks
• La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America
• M3MAD-Bench: Multi-Dimensional Evaluation of Multi-Agent Debate Across Domains and Modalities
• Reasoning as an Adaptive Defense for Safety
• SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
• Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusions
• Training Language Model to Critique for Better Refinement
• Complexity-aware fine-tuning
• Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
• OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs
• Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
• Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
• Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation
• MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages
• Inference-Time Reward Hacking in Large Language Models
• SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
• Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models
• Benchmarking the Pedagogical Knowledge of Large Language Models
• Shrinking the Generation-Verification Gap with Weak Verifiers
• Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution
• PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level
• Reviving Your MNEME: Predicting The Side Effects of LLM Unlearning and Fine-Tuning via Sparse Model Diffing
• PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction
• Alignment between Brains and AI: Evidence for Convergent Evolution across Modalities, Scales and Training Trajectories
• MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs
• Finance Language Model Evaluation (FLaME)
• NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models
• Mobile Application Review Summarization using Chain of Density Prompting
• MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application
• BOW: Training Language Models to Reason Over Plausible Next Words
• LocationReasoner: Evaluating LLMs on Real-World Site Selection Reasoning
• xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations
• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
• QFFT, Question-Free Fine-Tuning for Adaptive Reasoning
• SciDA: Scientific Dynamic Assessor of LLMs
• Overview of the NLPCC 2025 Shared Task: Gender Bias Mitigation Challenge
• Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
• TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
• Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index
• Training-free LLM Verification via Recycling Few-shot Examples
• VLM@school -- Evaluation of AI image understanding on German middle school knowledge
• VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
• Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning
• Building Models of Neurological Language
• dots.llm1 Technical Report
• Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
• RELIC: Evaluating Complex Reasoning via the Recognition of Languages In-Context
• Inference-Time Hyper-Scaling with KV Cache Compression
• Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning
• MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark
• Crosslingual Reasoning through Test-Time Scaling
• Training a Scientific Reasoning Model for Chemistry
• Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
• Aligning Large Language Models with Implicit Preferences from User-Generated Content
• Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement
• MiMo-VL Technical Report
• Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning
• Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem
• KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG
• ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities
• Adaptive Task Vectors for Large Language Models
• EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
• Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation
• MASTER: Enhancing Large Language Model via Multi-Agent Simulated Teaching
• FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes
• Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
• One Missing Piece for Open-Source Reasoning Models: A Dataset to Mitigate Cold-Starting Short CoT LLMs in RL
• Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models
• Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains
• Is Extending Modality The Right Path Towards Omni-Modality?
• ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation
• SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioning
• FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting
• Probing Neural Topology of Large Language Models
• Enhancing Clinical Multiple-Choice Questions Benchmarks with Knowledge Graph Guided Distractor Generation
• TeleEval-OS: Performance evaluations of large language models for operations scheduling
• BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluation
• LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text
• PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models
• Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
• DEEPQUESTION: Systematic Generation of Real-World Challenges for Evaluating LLMs Performance
• Semi-structured LLM Reasoners Can Be Rigorously Audited
• Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation
• ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases
• X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains
• Revisiting Uncertainty Estimation and Calibration of Large Language Models
• NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs
• Scalable Complexity Control Facilitates Reasoning Ability of LLMs
• Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures
• A Mathematical Framework for AI-Human Integration in Work
• AutoL2S: Auto Long-Short Reasoning for Efficient Large Language Models
• Advancing Expert Specialization for Better MoE
• Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data
• What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoning
• LLMs Judging LLMs: A Simplex Perspective
• Herd Behavior: Investigating Peer Influence in LLM-based Multi-Agent Systems
• Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator
• Reinforcing General Reasoning without Verifiers
• Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models
• Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning
• Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue History
• Don't Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models
• MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs
• Learning to Reason without External Rewards
• Multi-Agent Collaboration via Evolving Orchestration
• StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
• REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoning
• The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
• CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesis
• TTPA: Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluation
• Paying Alignment Tax with Contrastive Learning
• LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
• ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
• BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints
• SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data
• Knowledge Grafting of Large Language Models
• G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning
• TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
• CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
• Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL
• Large Language Models Do Multi-Label Classification Differently
• Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
• Global Optimization and Inference-Time Region Grafting for Agentic Workflows
• ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models
• RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs
• INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling
• EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs
• Relative Bias: A Comparative Framework for Quantifying Bias in LLMs
• SPaRC: A Spatial Pathfinding Reasoning Challenge
• Memorization or Reasoning? Exploring the Idiom Understanding of LLMs
• DuFFin: A Dual-Level Fingerprinting Framework for LLMs IP Protection
• SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning
• X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs
• MASLab: A Unified and Comprehensive Codebase for LLM-based Multi-Agent Systems
• Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey
• Scaling Physical Reasoning with the PHYSICS Dataset
• Systematic Evaluation of Machine-Generated Reasoning and PHQ-9 Labeling for Depression Detection Using Large Language Models
• Joint Flashback Adaptation for Forgetting-Resistant Instruction Tuning
• PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions
• Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought
• VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models
• TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning
• LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
• YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
• DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
• Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
• General-Reasoner: Advancing LLM Reasoning Across All Domains
• Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis
• Log-Augmented Generation: Scaling Test-Time Reasoning with Reusable Computation
• Reasoning Models Better Express Their Confidence
• WebNovelBench: Placing LLM Novelists on the Web Novel Distribution
• Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings
• J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization
• Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models
• Shadow-FT: Tuning Instruct Model via Training on Paired Base Model
• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
• SpatialLLM: From Multi-modality Data to Urban Spatial Intelligence
• CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Process
• On the Thinking-Language Modeling Gap in Large Language Models
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams
• Language Model Networks: Supervision-Efficient Learning through Dense Communication
• IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests
• Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
• AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning
• Model Merging in Pre-training of Large Language Models
• Evaluating the Logical Reasoning Abilities of Large Reasoning Models
• Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting
• Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
• Disentangling Reasoning and Knowledge in Medical Large Language Models
• A Systematic Analysis of Base Model Choice for Reward Modeling
• When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs
• Review-Instruct: A Review-Driven Multi-Turn Conversations Generation Method for Large Language Models
• WorldPM: Scaling Human Preference Modeling
• On the Evaluation of Engineering Artificial General Intelligence
• How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference
• Qwen3 Technical Report
• Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora
• Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results
• Evaluating LLM Metrics Through Real-World Capabilities
• What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs
• CoT-Core: Accelerating LLM Evaluation via CoT-Aware Coreset Selection
• Semantic Retention and Extreme Compression in LLMs: Can We Have Both?
• Towards Multi-Agent Reasoning Systems for Collaborative Expertise Delegation: An Exploratory Design Study
• MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining
• xGen-small Technical Report
• TeachArena: Are Language Agents Ready for Realistic Teaching Work?
• Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation
• R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
• When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
• Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale
• Apriel-1.5-OpenReasoner: RL Post-Training for General-Purpose and Efficient Reasoning
• TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering
• Measuring and Eliminating Refusals in Military Large Language Models
• Self-Ablating Transformers: More Interpretability, Less Sparsity
• Scaling Laws for Task-Specific LLM Distillation
• MiniMax Sparse Attention
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents
• Flaws in the LLM Automation Narrative
• Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese
• $OneMillion-Bench: How Far are Language Agents from Human Experts?
• Architecting Trust in Artificial Epistemic Agents
• Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference
• AI Evaluation Should Require Standardized Item-Level Data Releases
• Fully Open Meditron: An Auditable Pipeline for Clinical LLMs
• Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Models
• SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
• Auditing LLM Benchmarks with Item Response Theory
• BIRDS: Characterizing and Understanding Biodiversity Impact of Large Language Model Serving
• Training-Free Looped Transformers
• Scaling Embeddings Outperforms Scaling Experts in Language Models
• Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges
• SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture
• Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key
• Computational Reasoning of Large Language Models
• Can a Crow Hatch a Falcon? Lineage Matters in Predicting Large Language Model Performance
• Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving
• SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
• When are likely answers right? On Sequence Probability and Correctness in LLMs
• Life After Benchmark Saturation: A Case Study of CORE-Bench
• SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining
• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
• Learning is Forgetting: LLM Training As Lossy Compression
• PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering
• The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques
• Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs
• Large Language Model Agents Are Not Always Faithful Self-Evolvers
• Security Steerability is All You Need
• LAMBench: A Benchmark for Large Atomistic Models
• AutoJudge: Judge Decoding Without Manual Annotation
• Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
• AgentSPEX: An Agent SPecification and EXecution Language
• Towards Autonomous Mechanistic Reasoning in Virtual Cells
• Think in Sentences: Explicit Sentence Boundaries Enhance Language Model's Capabilities
• From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation
• Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation
• Reasoning Models Will Sometimes Lie About Their Reasoning
• RelayLLM: Efficient Reasoning via Collaborative Decoding
• Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR
• DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing
• Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective
• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
• Relational Priors as Convergence Pressure in LLM-Based Multi-Agent Systems
• Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
• The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity
• K-EXAONE 2.0 Technical Report
• A Blind Spot in Alignment: Quantifying Biosecurity Risks in Large Language Models
• Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
• T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models
• Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
• Learning to Reason under Off-Policy Guidance
• EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework
• gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
• LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception
• D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Model
• Human-Like Anaphor Resolution in Large Language Models
• MACRO: Markov Chain Routing of Transformer Layers
• Search2Skill: Skill Distillation Beyond Knowledge Boundaries Via Rubric-Based Reinforcement Learning
• Mean-Field Dynamics of Chain-of-Thought Reasoning in Large Language Models
• Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
• Benchmarking Vision Language Models on German Factual Data
• Assessing Judging Bias in Large Reasoning Models: An Empirical Study
• Guiding Reasoning in Small Language Models with LLM Assistance
• Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish
• Short-Path Prompting in LLMs: Analyzing Reasoning Instability and Solutions for Robust Performance
• Fast-Slow-Thinking: Complex Task Solving with Large Language Models
• Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning
• Large Language Models Could Be Rote Learners
• Playpen: An Environment for Exploring Learning Through Conversational Interaction
• Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
• Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
• GAAPO: Genetic Algorithmic Applied to Prompt Optimization
• FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion
• Self-Steering Language Models
• Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation
• From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
• SEA-LION: Southeast Asian Languages in One Network
• Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions
• Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
• Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors

Related

• MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoning
• MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
• MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models
• MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
• MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs