Reasoning with Sampling: Cutting at Decision Points

2026/05/28 by Felix Zhou, Anay Mehrotra, Quanquan C. Liu · 1 voice
Computer Science · Mathematics · #cs.AI #cs.CL #cs.LG #math.ST #stat.ML

paper · pdf · doi:10.48550/arxiv.2605.30327

Abstract

Frontier reasoning models are produced by posttraining base language models with reinforcement learning. Recent work has challenged this by showing that sampling from a sharpened version of the base model's distribution, a so-called power distribution, elicits comparable reasoning without additional training, curated datasets, or verifiers. However, making this method practical requires efficiently sampling from the power distribution. A sampler needs to "mix" to the power distribution, which necessitates moving between modes of the target distribution; intuitively, e.g., trying different reasoning strategies. The samplers proposed in prior works repeatedly select a "cut" position in the current reasoning trace uniformly at random and resample the suffix from that position onward. However, reasoning traces typically contain a few consequential decisions (e.g., the choice of proof strategy or algorithm), and we observe that a uniformly chosen cut tends to rewrite local details rather than revisit decision points. We introduce an algorithm (Entropy-Cut Metropolis-Hastings) that uses the base model's next-token entropy as a proxy to identify key decision points and resample from those positions. We empirically verify that entropy jumps are a useful proxy for decision points and, in a stylized model of reasoning, prove that our method's mixing time scales with the number of decisions in a trace rather than with the number of tokens, which can be much larger. Across MATH500, HumanEval, GPQA Diamond, and AIME26, our method consistently improves over baselines and RL-trained models.

Citations

• Reasoning with Sampling: Your Base Model is Smarter Than You Think
• Outcome-based Exploration for LLM Reasoning
• Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models
• Spurious Rewards: Rethinking Training Signals in RLVR
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• Maximizing Confidence Alone Improves Reasoning
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Entropy-based Exploration Conduction for Multi-step Reasoning
• Blink of an eye: a simple theory for feature localization in generative models
• Phi-4 Technical Report
• Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability
• Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Fast Inference from Transformers via Speculative Decoding
• On free energy barriers in Gaussian priors and failure of cold start MCMC for high-dimensional unimodal distributions
• OpenAI o1 System Card

Discussions

• A reasoning trace isn't uniform: a few tokens pick the strategy and the rest just execute it. Our sampler uses entropy to find those decision points and resample where the reasoning trace can branch. [bsky, 2 points, 0 comments]

Related

• Reasoning with Sampling: Your Base Model is Smarter Than You Think
• No Compute Left Behind: Rethinking Reasoning and Sampling with Masked Diffusion Models
• Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling
• EAGer: Entropy-Aware GEneRation for Adaptive Inference-Time Scaling
• Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study