Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling

2024/11/01 by Yiwen Ding, Ding, Yiwen, Zhiheng Xi +16 · 5 citations
Engineering · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reservoir Engineering and Simulation Methods

paper · pdf · doi:10.48550/arxiv.2411.00750

openalex publication_date 2024/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Self-improvement methods enable large language models (LLMs) to generate solutions themselves and iteratively train on filtered, high-quality rationales. This process proves effective and reduces the reliance on human supervision in LLMs' reasoning, but the performance soon plateaus. We delve into the process and find that models tend to over-sample on easy queries and under-sample on queries they have yet to master. As iterations proceed, this imbalance in sampling is exacerbated, leading to a long-tail distribution where solutions to difficult queries almost diminish. This phenomenon limits the performance gain of self-improving models. A straightforward solution is brute-force sampling to balance the distribution, which significantly raises computational costs. In this paper, we introduce Guided Self-Improvement (GSI), a strategy aimed at improving the efficiency of sampling challenging heavy-tailed data. It leverages Socratic-style guidance signals to help LLM reasoning with complex queries, reducing the exploration effort and minimizing computational overhead. Experiments on four models across diverse mathematical tasks show that GSI strikes a balance between performance and efficiency, while also being effective on held-out tasks.

Cited by

• DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
• AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress
• Counteracting Matthew Effect in Self-Improvement of LVLMs through Head-Tail Re-balancing
• Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning
• BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping

Related

• Importance Weighting Can Help Large Language Models Self-Improve
• Large Language Models Can Self-Improve
• The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding?
• From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
• Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling