To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization

2025/02/02 by Long Li, Wang, Haozhe, Chao Qu +10 · 14 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Educational Technology and Assessment #FOS: Computer and information sciences #Fuzzy Logic and Control Systems #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2502.00691

openalex publication_date 2025/02/02 · openalex created_date 2025/02/05 · openalex updated_date 2026/07/28

Abstract

Recent advances in mathematical problem-solving with language models (LMs) integrate chain-of-thought (CoT) reasoning and code execution to harness their complementary strengths. However, existing hybrid frameworks exhibit a critical limitation: they depend on externally dictated instructions or rigid code-integration templates, lacking metacognitive awareness -- the capacity to dynamically evaluate intrinsic capabilities and autonomously determine when and how to integrate tools. This rigidity motivates our study of autonomous code integration, enabling models to adapt tool-usage strategies as their reasoning abilities evolve during training. While reinforcement learning (RL) shows promise for boosting LLM reasoning at scale (e.g., DeepSeek-R1), we demonstrate its inefficiency in learning autonomous code integration due to inadequate exploration of the vast combinatorial space of CoT-code interleaving patterns. To address this challenge, we propose a novel Expectation-Maximization (EM) framework that synergizes structured exploration (E-step) with off-policy RL optimization (M-step), creating a self-reinforcing cycle between metacognitive tool-use decisions and evolving capabilities. Experiments reveal our method achieves superior results through improved exploration. Notably, our 7B model improves over 11% on MATH500 and 9.4% on AIME without o1-like CoT.

Cited by

• Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation
• UniRel: Relation-Centric Knowledge Graph Question Answering with RL-Tuned LLM Reasoning
• CogDoc: Towards Unified thinking in Documents
• From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning
• STAR-PólyaMath: Multi-Agent Reasoning under Persistent Meta-Strategic Supervision
• Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing
• UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning
• StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
• Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
• Reverse-Engineered Reasoning for Open-Ended Generation
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
• Attributes as Textual Genes: Leveraging LLMs as Genetic Algorithm Simulators for Conditional Synthetic Data Generation
• VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
• Fast, Slow, and Tool-augmented Thinking for LLMs: A Review

Related

• Towards Effective Code-Integrated Reasoning
• How Do Humans Write Code? Large Models Do It the Same Way Too
• CoRT: Code-integrated Reasoning within Thinking
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
• Code-enabled language models can outperform reasoning models on diverse tasks