GPQA: A Graduate-Level Google-Proof Q&A Benchmark

2023/11/20 by David B. Rein, David Rein, Betty Li Hou +14 · 12 voices · 680 citations
Computer Science · #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2311.12022

Abstract

We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.

Cited by

• Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
• HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery
• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
• HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference
• Attention Residuals
• Explainable Model Routing for Agentic Workflows
• Wrong and More Confident: A Field Experiment on Large Language Models Taking a Graduate Economics Exam
• Delegation Intelligence in Deep Search: A Controllable Framework for Disentangled Capability Diagnosis
• Semantic-Enhanced Automatic Refinement of Architecture Recovery Results Using LLMs
• E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
• SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows
• Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
• Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias
• Beyond Block Boundaries: Multi-Block Editing for Diffusion Large Language Models
• Procedural Knowledge at Scale Improves Reasoning
• Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact
• Controllable LLM Reasoning via Sparse Autoencoder-Based Steering
• Entropy Sentinel: Probing Entropy Traces for LLM Monitoring
• QMBench: A Research Level Benchmark for Quantum Materials Research
• Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards
• Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model
• Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
• Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty
• Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks
• LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamics
• Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
• Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
• MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
• Xiaomi MiMo-VL-Miloco Technical Report
• DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
• Sigma-MoE-Tiny Technical Report
• ToolForge: A Data Synthesis Pipeline for Multi-Hop Search without Real-World APIs
• Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning
• Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams
• T5Gemma 2: Seeing, Reading, and Understanding Longer
• Estimating problem difficulty without ground truth using Large Language Model comparisons
• Grammar Search for Multi-Agent Systems
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value
• Textual Gradients are a Flawed Metaphor for Automatic Prompt Optimization
• ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
• AIR: Post-training Data Selection for Reasoning via Attention Head Influence
• AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
• M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization
• Wait, Wait, Wait... Why Do Reasoning Models Loop?
• SIGMA: An AI-Empowered Training Stack on Early-Life Hardware
• Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
• Coupled Variational Reinforcement Learning for Language Model General Reasoning
• AI Benchmark Democratization and Carpentry
• Rethinking Expert Trajectory Utilization in LLM Post-training
• Benchmarking the Generality of Vision-Language-Action Models
• Progress over Points: Reframing LM Benchmarks Around Scientific Objectives
• Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
• SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing
• BAMBO: Construct Ability and Efficiency LLM Pareto Set via Bayesian Adaptive Multi-objective Block-wise Optimization
• LLaDA2.0: Scaling Up Diffusion Language Models to 100B
• CONCUR: A Framework for Continual Constrained and Unconstrained Routing
• EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce
• Towards a Science of Scaling Agent Systems
• Single-Agent Scaling Fails Multi-Agent Intelligence: Towards Foundation Models with Native Multi-Agent Intelligence
• Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
• Uncovering Competency Gaps in Large Language Models and Their Benchmarks
• Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models
• A Latent Variable Framework for Scaling Laws in Large Language Models
• Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy
• Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment
• SIMA 2: A Generalist Embodied Agent for Virtual Worlds
• Qwen3.5-Omni Technical Report
• The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems
• Model Whisper: Steering Vectors Unlock Large Language Models' Potential in Test-time
• On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference
• Distance Is All You Need: Radial Dispersion for Uncertainty Estimation in Large Language Models
• PARC: An Autonomous Self-Reflective Coding Agent for Robust Execution of Long-Horizon Tasks
• Exploring the Potential and Limitations of Large Language Models for Novice Program Fault Localization
• Lumos: Let there be Language Model System Certification
• MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
• Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules
• SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment
• From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• Guided Self-Evolving LLMs with Minimal Human Supervision
• When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers
• Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoning
• LeechHijack: Covert Computational Resource Exploitation in Intelligent Agent Systems
• The Art of Scaling Test-Time Compute for Large Language Models
• Neural steering vectors reveal dose and exposure-dependent impacts of human-AI relationships
• Rectifying LLM Thought from Lens of Optimization
• Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
• SUPERChem: A Multimodal Reasoning Benchmark in Chemistry
• AI-Trader: Benchmarking Autonomous Agents in Real-Time Financial Markets
• Lightweight Latent Reasoning for Narrative Tasks
• EDIT: Early Diffusion Inference Termination for dLLMs Based on Dynamics of Training Gradients
• Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction
• ORION: Teaching Language Models to Reason Efficiently in the Language of Thought
• A Rosetta Stone for AI Benchmarks
• Revisiting Generalization Across Difficulty Levels: It's Not So Easy
• AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models
• Qwen3-VL Technical Report
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
• Structured Prompting Enables More Robust Evaluation of Language Models
• Knowing How to Edit: Reliable Evaluation Signals for Diagnosing and Optimizing Prompts at Query Level
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• Latent Collaboration in Multi-Agent Systems
• Training-Free Active Learning Framework in Materials Science with Large Language Models
• LLMs-Powered Real-Time Fault Injection: An Approach Toward Intelligent Fault Test Cases Generation
• Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models
• Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost
• Reasoning With a Star: A Heliophysics Dataset and Benchmark for Agentic Scientific Reasoning
• AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
• Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discovery
• SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization
• MultiGA: Leveraging Multi-Source Seeding in Genetic Algorithms
• E³-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
• Asking LLMs to Verify First is Almost Free Lunch
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
• Monte Carlo Expected Threat (MOCET) Scoring
• Large Language Model-Based Reward Design for Deep Reinforcement Learning-Driven Autonomous Cyber Defense
• Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
• JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation
• Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
• EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
• Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
• Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
• ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning
• P1: Mastering Physics Olympiads with Reinforcement Learning
• Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions
• BeDiscovER: The Benchmark of Discourse Understanding in the Era of Reasoning Language Models
• BARD: budget-aware reasoning distillation
• Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
• Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
• Optimal Self-Consistency for Efficient Reasoning with Large Language Models
• Better LLM Reasoning via Dual-Play
• MetaGDPO: Alleviating Catastrophic Forgetting with Metacognitive Knowledge through Group Direct Preference Optimization
• PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
• From Single to Societal: Analyzing Persona-Induced Bias in Multi-Agent Interactions
• Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation
• Virtual Width Networks
• On the Measure of a Model: From Intelligence to Generality
• When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
• ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference
• AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment
• Seer Self-Consistency: Advance Budget Estimation for Adaptive Test-Time Scaling
• Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models
• Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models
• SPEAR-MM: Selective Parameter Evaluation and Restoration via Model Merging for Efficient Financial LLM Adaptation
• SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning
• DynaAct: Large Language Model Reasoning with Dynamic Action Spaces
• SERL: Self-Examining Reinforcement Learning on Open-Domain
• Investigating CoT Monitorability in Large Reasoning Models
• RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Services
• Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives
• S-DAG: A Subject-Based Directed Acyclic Graph for Multi-Agent Heterogeneous Reasoning
• MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
• CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Models
• MONICA: Real-Time Monitoring and Calibration of Chain-of-Thought Sycophancy in Large Reasoning Models
• What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Models
• Chain-of-Thought as a Lens: Evaluating Structured Reasoning Alignment between Human Preferences and Large Language Models
• Tiny Model, Big Logic: Diversity-Driven Optimization Elicits Large-Model Reasoning Ability in VibeThinker-1.5B
• Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
• OckBench: Measuring the Efficiency of LLM Reasoning
• If I Could Turn Back Time: Temporal Reframing as a Historical Reasoning Task for LLMs
• Motif 2 12.7B technical report
• Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm
• An MLCommons Scientific Benchmarks Ontology
• Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics
• Reusing Pre-Training Data at Test Time is a Compute Multiplier
• Batch Prompting Suppresses Overthinking Reasoning Under Constraint: How Batch Prompting Suppresses Overthinking in Reasoning Models
• NVIDIA Nemotron Nano V2 VL
• LiveTradeBench: Seeking Real-World Alpha with Large Language Models
• CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Sampling
• A Hierarchical Multi-Agent System for Autonomous Discovery in Geoscientific Data Archives
• Zero-shot data citation function classification using transformer-based large language models (LLMs)
• Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Series
• POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarios
• Apriel-H1: Towards Efficient Enterprise Reasoning Models
• CGES: Confidence-Guided Early Stopping for Efficient and Accurate Self-Consistency
• Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining
• Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning
• Shorter but not Worse: Frugal Reasoning via Easy Samples as Length Regularizers in Math RLVR
• DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching
• SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding
• Word Salad Chopper: Reasoning Models Waste A Ton Of Decoding Budget On Useless Repetitions, Self-Knowingly
• Culture Cartography: Mapping the Landscape of Cultural Knowledge
• Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoning
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains
• Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
• Reasoning Models Sometimes Output Illegible Chains of Thought
• A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios
• LongCat-Flash-Omni Technical Report
• SIGMA: Search-Augmented On-Demand Knowledge Integration for Agentic Mathematical Reasoning
• H-FA: A Hybrid Floating-Point and Logarithmic Approach to Hardware Accelerated FlashAttention
• Value Drifts: Tracing Value Alignment During LLM Post-Training
• The End of Manual Decoding: Towards Truly End-to-End Language Models
• EdgeRunner 20B: Military Task Parity with GPT-5 while Running on the Edge
• Think Outside the Policy: In-Context Steered Policy Optimization
• From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learning
• Do LLMs Signal When They're Right? Evidence from Neuron Agreement
• RCScore: Quantifying Response Consistency in Large Language Models
• Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math
• QuantumBench: A Benchmark for Quantum Problem Solving
• e1: Learning Adaptive Control of Reasoning Effort
• Gaperon: A Peppered English-French Generative Language Model Suite
• Are Language Models Efficient Reasoners? A Perspective from Logic Programming
• Zero Reinforcement Learning Towards General Domains
• Why Public Service AI Governance Frameworks Risk Failing in the Age of General-Purpose AI: Lessons from Policing
• RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement
• When is Routing Meaningful? Diversity and Robustness in Language Model Societies
• Weak-to-Strong On-Policy Distillation
• When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
• PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning
• Will Scaling Improve Social Simulation with LLMs?
• SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
• The Capability Paradox: How Smarter Auditors Make Multi-Agent Systems Less Secure
• Guess-Verify-Refine: Data-Aware Top-K for Sparse-Attention Decoding on Blackwell via Temporal Correlation
• Arcee Trinity Large Technical Report
• Robustness as an Emergent Property of Task Performance
• Ministral 3
• Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process
• Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
• RAVR: Reference-Answer-guided Variational Reasoning for Large Language Models
• Automating Benchmark Design
• AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis
• APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training
• ChessQA: Evaluating Large Language Models for Chess Understanding
• Fortytwo: Swarm Inference with Peer-Ranked Consensus
• Multi-Agent Evolve: LLM Self-Improve through Co-evolution
• Unleashing Diverse Thinking Modes in LLMs through Multi-Agent Collaboration
• LimRank: Less is More for Reasoning-Intensive Information Reranking
• Knocking-Heads Attention
• AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment
• FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
• Scalable Oversight via Partitioned Human Supervision
• Modeling Hierarchical Thinking in Large Reasoning Models
• Risk Management for Mitigating Benchmark Failure Modes: BenchRisk
• When Models Outthink Their Safety: Mitigating Self-Jailbreak in Large Reasoning Models with Chain-of-Guardrails
• Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
• The Universal Landscape of Human Reasoning
• A Coherence-Based Measure of AGI
• What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation
• Teaching Language Models to Reason with Tools
• Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values
• The Reasoning Lingua Franca: A Double-Edged Sword for Multilingual AI
• ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows
• Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training
• Can Confidence Estimates Decide When Chain-of-Thought Is Necessary for LLMs?
• SmartSwitch: Advancing LLM Reasoning by Overcoming Underthinking via Promoting Deeper Thought Exploration
• A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation
• An Expert-grounded benchmark of General Purpose LLMs in LCA
• Teaming LLMs to Detect and Mitigate Hallucinations
• The Zero-Step Thinking: An Empirical Study of Mode Selection as Harder Early Exit in Reasoning Models
• DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference
• Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
• Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
• EffiReasonTrans: RL-Optimized Reasoning for Code Translation
• A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
• WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection
• Reasoning Language Model Inference Serving Unveiled: An Empirical Study
• CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
• Mapping Post-Training Forgetting in Language Models at Scale
• Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
• Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning
• JT-Safe: Intrinsically Enhancing the Safety and Trustworthiness of LLMs
• A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• Neuronal Group Communication for Efficient Neural representation
• Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
• ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoning
• LLMs Judge Themselves: A Game-Theoretic Framework for Human-Aligned Evaluation
• MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs
• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning
• SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradients
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
• Reasoning with Sampling: Your Base Model is Smarter Than You Think
• Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following
• Qwen3Guard Technical Report
• LiteStage: Latency-aware Layer Skipping for Multi-stage Reasoning
• Budget-aware Test-time Scaling via Discriminative Verification
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems
• CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoning
• Selective Adversarial Attacks on LLM Benchmarks
• Schema for In-Context Learning
• Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering
• Dr.LLM: Dynamic Layer Routing in LLMs
• Demystifying Hybrid Thinking: Can LLMs Truly Switch Between Think and No-Think?
• Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems
• ResearStudio: A Human-Intervenable Framework for Building Controllable Deep-Research Agents
• MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
• ThinkPilot: Steering Reasoning Models via Automated Think-prefixes Optimization
• HoneyBee: Data Recipes for Vision-Language Reasoners
• CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research
• LLM Reasoning for Machine Translation: Synthetic Data Generation over Thinking Tokens
• Deep Research Brings Deeper Harm
• Are Large Reasoning Models Interruptible?
• Demystifying Reinforcement Learning in Agentic Reasoning
• ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problems
• A²FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning
• Enhancing Long Chain-of-Thought Reasoning through Multi-Path Plan Aggregation
• Information-Preserving Reformulation of Reasoning Traces for Antidistillation
• From to : Multidimensional Supervision of Reasoning Process for LLM Optimization
• LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences
• Neural Weight Compression for Language Models
• Can Tool-Integrated Reinforcement Learning Generalize Across Diverse Domains?
• EAGER: Entropy-Aware GEneRation for Adaptive Inference-Time Scaling
• DND: Boosting Large Language Models with Dynamic Nested Depth
• DeepResearchGuard: Deep Research with Open-Domain Evaluation and Multi-Stage Guardrails for Safety
• Revisiting Model Interpolation for Efficient Reasoning
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
• ExGRPO: Learning to Reason from Experience
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration
• Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning
• Merlin's Whisper: Enabling Efficient Reasoning in LLMs via Black-box Adversarial Prompting
• Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning
• Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
• The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities
• Stop When Enough: Adaptive Early-Stopping for Chain-of-Thought Reasoning
• Brick: Spatial Capability Routing for the Mixture-of-Models (MoM) Paradigm
• SkillOS: Learning Skill Curation for Self-Evolving Agents
• The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More
• PostTrainBench: Can LLM Agents Automate LLM Post-Training?
• DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
• Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness
• Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
• InteractScience: Programmatic and Visually-Grounded Evaluation of Interactive Scientific Demonstration Code Generation
• ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability
• Constraints-of-Thought: A Framework for Constrained Reasoning in Language-Model-Guided Search
• HES-SQL: Hybrid Reasoning for Efficient Text-to-SQL with Structural Skeleton Guidance
• DualResearch: Entropy-Gated Dual-Graph Retrieval for Answer Reconstruction
• RegexPSPACE: A Benchmark for Evaluating LLM Reasoning on PSPACE-complete Regex Problems
• On the Provable Performance Guarantee of Efficient Reasoning Models
• KORMo: Korean Open Reasoning Model for Everyone
• Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs
• How Reliable is Language Model Micro-Benchmarking?
• CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards
• FlowSearch: Advancing deep research with dynamic structured knowledge flow
• DeepPrune: Parallel Scaling without Inter-trace Redundancy
• ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping
• Guided Star-Shaped Masked Diffusion
• The Alignment Waltz: Jointly Training Agents to Collaborate for Safety
• LLMs Learn to Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions
• ARM2: Adaptive Reasoning Model with Vision Understanding and Executable Code
• Upfront Chain-of-Thought: A Cooperative Framework for Chain-of-Thought Compression
• Iterated Agent for Symbolic Regression
• Learning on the Job: An Experience-Driven Self-Evolving Agent for Long-Horizon Tasks
• Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
• Measuring and Mitigating Identity Bias in Multi-Agent Debate via Anonymization
• More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning
• Native Hybrid Attention for Efficient Sequence Modeling
• Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languages
• SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
• Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
• SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering
• Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression
• Mid-Training of Large Language Models: A Survey
• Gold-Switch: Training-Free Superposition of Slow- and Fast- Thinking LLMs
• Inefficiencies of Meta Agents for Agent Design
• Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
• AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning
• The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
• What MLLMs Learn about When they Learn about Multimodal Reasoning
• AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding
• Online Rubrics Elicitation from Pairwise Comparisons
• h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
• Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoning
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
• TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning
• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• Influence Functions for Efficient Data Selection in Reasoning
• ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
• Luth: Efficient French Specialization for Small Language Models and Cross-Lingual Transfer
• ARM: Discovering Agentic Reasoning Modules for Generalizable Multi-Agent Systems
• Membership Inference Attacks on Tokenizers of Large Language Models
• ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
• MixReasoning: Switching Modes to Think
• Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering
• Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
• InvThink: Premortem Reasoning for Safer Language Models
• BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
• CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers
• Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models
• Slm-mux: Orchestrating small language models for reasoning
• SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs
• Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
• Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
• Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
• Distilling Reasoning into Student LLMs: Local Naturalness for Selecting Teacher Data
• Measuring Language Model Hallucinations Through Distributional Correctness
• COSMO-RL: Towards Trustworthy LMRMs via Joint Safety and Stability
• Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models
• MedReflect: Teaching Medical LLMs to Self-Improve via Reflective Correction
• GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Time
• Self-Reflective Generation at Test Time
• Uncovering the Computational Ingredients of Human-Like Representations in LLMs
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
• On Predictability of Reinforcement Learning Dynamics for Large Language Models
• ThinkBrake: Mitigating Overthinking in Tool Reasoning
• Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs
• Rethinking Reward Models for Multi-Domain Test-Time Scaling
• In-Place Feedback: Reliable Refinement for Multi-Turn Expert-LLM Collaboration
• Generalized Parallel Scaling with Interdependent Generations
• Rethinking Thinking Tokens: LLMs as Improvement Operators
• Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
• Learning Compact Representations of LLM Abilities via Item Response Theory
• Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
• Stochastic Self-Organization in Multi-Agent Systems
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
• Training Large Language Models To Reason In Parallel With Global Forking Tokens
• DecepChain: Inducing Deceptive Reasoning in Large Language Models
• On The Fragility of Benchmark Contamination Detection in Reasoning Models
• Towards Ecologically Valid LLM Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners
• Are Robust LLM Fingerprints Adversarially Robust?
• Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark
• Entropy After ⟨ /Think ⟩ for reasoning model early exiting
• Revealing the Power of Post-Training for Small Language Models via Knowledge Distillation
• Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
• RoRecomp: Enhancing Reasoning Efficiency via Rollout Response Recomposition in Reinforcement Learning
• Overthinking Reduction with Decoupled Rewards and Curriculum Data Scheduling
• Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
• Collaborative Compression for Large-Scale MoE Deployment on Edge
• Nudging the Boundaries of LLM Reasoning
• Unspoken Hints: Accuracy Without Acknowledgement in LLM Reasoning
• TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture
• Diversity-Incentivized Exploration for Versatile Reasoning
• Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training
• Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced Performance Gap
• SimulRAG: Simulator-based RAG for Grounding LLMs in Long-form Scientific QA
• RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMs
• UniAPL: A Unified Adversarial Preference Learning Framework for Instruct-Following
• The Era of Real-World Human Interaction: RL from User Conversations
• CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• SecInfer: Preventing Prompt Injection via Inference-time Scaling
• TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Models
• LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space
• AdaThink-Med: Medical Adaptive Thinking with Uncertainty-Guided Length Calibration
• Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settings
• Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention
• Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis
• SpecExit: Accelerating Large Reasoning Model via Speculative Exit
• Learning to Ponder: Adaptive Reasoning in Latent Space
• Your thoughts tell who you are: Characterize the reasoning patterns of LRMs
• Watermarking Diffusion Language Models
• ARS: Adaptive Reasoning Suppression for Efficient Large Reasoning Language Models
• Unlocking Zero-Shot Geospatial Reasoning via Indirect Rewards
• Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Models
• ReasonCACHE: Teaching LLMs To Reason Without Weight Updates
• Beyond Magic Words: Sharpness-Aware Prompt Evolving for Robust Large Language Models with TARE
• Sequential Diffusion Language Models
• HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs
• Tequila: Trapping-free Ternary Quantization for Large Language Models
• Beyond the Exploration-Exploitation Trade-off: A Hidden State Approach for LLM Reasoning in RLVR
• Timber: Training-free Instruct Model Refining with Base via Effective Rank
• Efficient Turing Machine Simulation with Transformers
• Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks
• EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance
• Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs
• A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Models
• Zero-shot evaluation reveals limitations of single-cell foundation models
• p-less Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding
• From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
• Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs
• Multiplayer Nash Preference Optimization
• Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Models
• d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
• JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
• HEART: Emotionally-driven test-time scaling of Language Models
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• Variational Reasoning for Language Models
• InfiR2: A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models
• REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Model
• Representing LLMs in Prompt Semantic Task Space
• Meta-Awareness Enhances Reasoning Models: Self-Alignment Reinforcement Learning
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
• COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
• The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Merging
• No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
• Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
• Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
• RLP: Reinforcement as a Pretraining Objective
• Hallucination reduction with CASAL: Contrastive Activation Steering For Amortized Learning
• Evidence for Limited Metacognition in LLMs
• Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning
• Best-of-∞ -- Asymptotic Performance of Test-Time Compute
• Distilling Many-Shot In-Context Learning into a Cheat Sheet
• On Theoretical Interpretations of Concept-Based In-Context Learning
• Who's Laughing Now? An Overview of Computational Humour Generation and Explanation
• MARS: toward more efficient multi-agent collaboration for LLM reasoning
• Thinking Augmented Pre-training
• TRUEBench: Can LLM Response Meet Real-world Constraints as Productivity Assistant?
• RAR²: Retrieval-Augmented Medical Reasoning via Thought-Driven Retrieval
• SKYLENAGE Technical Report: Mathematical Reasoning and Contest-Innovation Benchmarks for Multi-Level Math Evaluation
• The Conductor and the Engine: A Path Towards Co-Designed Reasoning
• Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation
• LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
• SKIMIX: Multi-Agent Harness-Time Scaling with Skill Mixture for Dynamic Harness Engineering
• Reasoning Consensus: Structural Ensembling of LLM Reasoning via Weighted DAG Aggregation
• Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game
• Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
• Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models in Commonsense Reasoning
• Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks
• Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset
• Divergence Decoding: Training-Free Capability Fusion
• Multi-Head Attention Residuals
• Reinforcement Learning Towards Broadly and Persistently Beneficial Models
• SuperThoughts: Reasoning Tokens in Superposition
• Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models
• The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
• Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
• Benchmark for Assessing Olfactory Perception of Large Language Models
• The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation From Recognition to Reasoning
• Evaluating large language models for accuracy incentivizes hallucinations
• Proximal Supervised Fine-Tuning
• DRQA: Dynamic Reasoning Quota Allocation for Controlling Overthinking in Reasoning Large Language Models
• Reinforcement Learning on Pre-Training Data
• Spacer: Towards Engineered Scientific Inspiration
• Introducing LongCat-Flash-Thinking: A Technical Report
• What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT
• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
• Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoning
• Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Process
• The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
• Evaluating the Safety and Skill Reasoning of Large Reasoning Models Under Compute Constraints
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• Understanding Post-Training Structural Changes in Large Language Models
• Qwen3-Omni Technical Report
• Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
• Stop Spinning Wheels: Mitigating LLM Overthinking via Mining Patterns for Early Reasoning Exit
• seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs
• SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamics
• RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation
• Benchmarking and Improving LLM Robustness for Personalized Generation
• KAIO: A Collection of More Challenging Korean Questions
• MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in Large Language Models
• Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
• Self-Improvement of Language Models by Post-Training on Multi-Agent Debate
• DSFT: Inspiring Diffusion Large Language Models to Comprehend Mathematical and Logical Patterns
• Early Stopping Chain-of-thoughts in Large Language Models
• Slim-SC: Thought Pruning for Efficient Scaling with Self-Consistency
• SteeringSafety: A Systematic Safety Evaluation Framework of Representation Steering in LLMs
• When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuning
• Leveraging Large Language Models to Effectively Generate Visual Data for Canine Musculoskeletal Diagnoses
• Root Cause Analysis of Radiation Oncology Incidents Using Large Language Models
• MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUs
• MALLM: Multi-Agent Large Language Models Framework
• Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces as Retrieval Sources for Domain Adaptation of Small Language Models
• Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
• Merge-of-Thought Distillation
• PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactions
• Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding
• Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yet
• RL Fine-Tuning Heals OOD Forgetting in SFT
• Orchestrator: Active Inference for Multi-Agent Systems in Long-Horizon Tasks
• Hunyuan-MT Technical Report
• Towards a Unified View of Large Language Model Post-Training
• On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
• Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
• Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish
• Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving
• Implicit Reasoning in Large Language Models: A Comprehensive Survey
• JudgeAgent: Knowledge-wise and Dynamic LLM Evaluation with Agent-as-Interviewer
• Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing
• EigenBench: A Comparative Behavioral Measure of Value Alignment
• Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation
• KoBLEX: Open Legal Question Answering with Multi-hop Reasoning
• Dream-Coder 7B: An Open Diffusion Language Model for Code
• DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks
• Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
• Throttling Web Agents Using Reasoning Gates
• LongCat-Flash Technical Report
• Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
• ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics
• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
• PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chains
• Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning
• From Canonical to Complex: Benchmarking LLM Capabilities in Undergraduate Thermodynamics
• BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning
• AI Reasoning Models for Problem Solving in Physics
• Intern-S1: A Scientific Multimodal Foundation Model
• TCIA: A Task-Centric Instruction Augmentation Method for Instruction Finetuning
• rStar2-Agent: Agentic Reasoning Technical Report
• Diffusion Language Models Know the Answer Before Decoding
• Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?
• Lean Meets Theoretical Computer Science: Scalable Synthesis of Theorem Proving Challenges in Formal-Informal Pairs
• Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
• Test-time Corpus Feedback: From Retrieval to RAG
• CAC-CoT: Connector-Aware Compact Chain-of-Thought for Efficient Reasoning Data Synthesis Across Dual-System Cognitive Tasks
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Hermes 4 Technical Report
• Dream 7B: Diffusion Large Language Models
• Deep Think with Confidence
• LongRecall: A Structured Approach for Robust Recall Evaluation in Long-Form Text
• Long Chain-of-Thought Reasoning Across Languages
• Don't Think Twice! Over-Reasoning Impairs Confidence Calibration
• Zero-knowledge LLM hallucination detection and mitigation through fine-grained cross-model consistency
• Your Reward Function for RL is Your Best PRM for Search: Unifying RL and Search-Based TTS
• Lexical Hints of Accuracy in LLM Reasoning Chains
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
• Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QA
• Input-Time Scaling
• Datarus-R1: An Adaptive Multi-Step Reasoning LLM for Automated Data Analysis
• G²RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
• PC-Sampler: Position-Aware Calibration of Decoding Bias in Masked Diffusion Models
• RAJ-PGA: Reasoning-Activated Jailbreak and Principle-Guided Alignment Framework for Large Reasoning Models
• DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoning
• Deep Research: A Survey of Autonomous Research Agents
• Reinforcement Learning with Rubric Anchors
• Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
• The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution
• Mitigating Jailbreaks with Intent-Aware LLMs
• LARC: Towards Human-level Constrained Retrosynthesis Planning through an Agentic Framework
• STEM: Efficient Relative Capability Evaluation of LLMs through Structured Transition Samples
• CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention
• Thinking Inside the Mask: In-Place Prompting in Diffusion LLMs
• Efficient Forward-Only Data Valuation for Pretrained LLMs and VLMs
• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
• Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation
• Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
• STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports
• Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
• User-centric Subjective Leaderboard by Customizable Reward Modeling
• Search-Time Data Contamination
• SinLlama -- A Large Language Model for Sinhala
• ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs
• InternBootcamp Technical Report: Boosting LLM Reasoning with Verifiable Task Scaling
• Retrospective Sparse Attention for Efficient Long-Context Generation
• Jinx: Unlimited LLMs for Probing Alignment Failures
• Evaluating Large Language Models as Expert Annotators
• ThinkTuning: Instilling Cognitive Reflections without Distillation
• Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
• AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance
• Context Misleads LLMs: The Role of Context Filtering in Maintaining Safe Alignment of LLMs
• MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams
• gpt-oss-120b & gpt-oss-20b Model Card
• Synthesizing scientific literature with retrieval-augmented language models
• SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
• Efficient Reasoning for Large Reasoning Language Models via Certainty-Guided Reflection Suppression
• MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
• IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards
• ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments
• Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward
• LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking
• Estimating Worst-Case Frontier Risks of Open-Weight LLMs
• RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Prior
• Thinking with Nothinking Calibration: A New In-Context Learning Paradigm in Reasoning Large Language Models
• CTTS: Collective Test-Time Scaling
• CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment
• Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following
• Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning
• Test-time Prompt Intervention
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction
• Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning
• A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges
• BioDisco: Multi-agent hypothesis generation with dual-mode evidence, iterative feedback and temporal evaluation
• Prompting Science Report 3: I'll pay you or I'll kill you -- but will you care?
• RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
• Cascaded Information Disclosure for Generalized Evaluation of Problem Solving Capabilities
• CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks
• TextQuests: How Good are LLMs at Text-Based Video Games?
• Unveiling Super Experts in Mixture-of-Experts Large Language Models
• BAR Conjecture: the Feasibility of Inference Budget-Constrained LLM Services with Authenticity and Reasoning
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner
• Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity

Discussions

• We do have data on how human PhDs do on the GPQA diamond. No need to be sarcastic. arxiv.org/abs/2311.12022 [bsky, 4 points, 1 comments]
• You're not. It's a bad quip (the one in question is often guilty of this and is more carnival barker than anything else), but he's referring to performance on a specific benchmark that tests responses [bsky, 3 points, 0 comments]
• Sure! I'm sitting down to lunch in the middle of a Vancouver so this is not from my notes but from memory and search. Google biases Arxiv, but some of the work is reviewed now. Notable project: [bsky, 3 points, 2 comments]
• Cela étant, je pense que le fait d’évoquer les PhD provient spécifiquement des résultats de chatgpt à ce benchmark arxiv.org/pdf/2311.12022 Qui est une suite de question scientifiques faites pour que [bsky, 2 points, 1 comments]
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark [lobsters, 2 points, 0 comments]
• GPQA Diamond consists of 198 graduate-level multiple-choice questions in biology, chemistry, and physics. Questions were developed and vetted by experts either pursuing or holding a PhD in the relevan [bsky, 1 points, 1 comments]
• Chatbots are just bare intuitive algorithms without the necessary symbolic wrappers (memory, deliberation symbols, etc), so yes they make odd mistakes. But the “PhD level” marketing means something [bsky, 1 points, 1 comments]
• arxiv.org/abs/2311.12022 [bsky, 1 points, 1 comments]
• And a lot of newer benchmarks have been designed to avoid things like data contamination, e.g. the PhD-level science benchmark questions were designed to be ungoogleable, and expert test takers had fu [bsky, 0 points, 1 comments]
• GPQA (Graduate-level, Google-Proof Q&A) is one of the toughest benchmarks for #LLMs, testing real scientific reasoning, not memorization (arxiv.org/pdf/2311.12022). If a model does well here, it’s thi [bsky, 0 points, 1 comments]
• btw, experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy on the GPQA (74% when discounting clear mistakes the experts identified in retrospect) arxiv.org/abs/2311.12 [bsky, 0 points, 1 comments]
• The last LLM test I looked into was the GPQA Diamond, which might be better but I think has it's own incentive problems. arxiv.org/abs/2311.12022 That dataset is was generated by having PhD's on Upwo [bsky, 0 points, 1 comments]

Related

• GAIA: a benchmark for General AI Assistants
• Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces as Retrieval Sources for Domain Adaptation of Small Language Models
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Scientific QA System with Verifiable Answers
• UQ: Assessing Language Models on Unsolved Questions