ReCode: Updating Code API Knowledge with Reinforcement Learning

2025/06/25 by Haoze Wu, Wu, Haoze, Yunzhi Yao +5 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Software Engineering (cs.SE) #Software Engineering Research #Software System Performance and Reliability #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2506.20495

openalex publication_date 2025/06/25 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) exhibit remarkable code generation capabilities but falter when adapting to frequent updates in external library APIs. This critical limitation, stemming from reliance on outdated API knowledge from their training data, even with access to current documentation, impedes reliable code generation in dynamic environments. To tackle this issue, we propose ReCode (rule-based Reinforcement learning for Code Update), a novel framework that mimics human programmer adaptation to API changes. Specifically, we construct a dataset of approximately 2,000 data entries to train the LLMs to perform version migration based on updated information. Then, we introduce a modified string similarity metric for code evaluation as the reward for reinforcement learning. Our experiments demonstrate that ReCode substantially boosts LLMs' code generation performance in dynamic API scenarios, especially on the unseen CodeUpdateArena task. Crucially, compared to supervised fine-tuning, ReCode has less impact on LLMs' general code generation abilities. We apply ReCode on various LLMs and reinforcement learning algorithms (GRPO and DAPO), all achieving consistent improvements. Notably, after training, Qwen2.5-Coder-7B outperforms that of the 32B parameter code instruction-tuned model and the reasoning model with the same architecture. Code is available at https://github.com/zjunlp/ReCode.

Citations

• The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
• WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents
• Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions
• SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• ToolRL: Reward is All Tool Learning Needs
• SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL
• MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning
• SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learning
• Video-R1: Reinforcing Video Reasoning in MLLMs
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs
• A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
• GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• CodeUpdateArena: Benchmarking Knowledge Editing on API Updates
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
• CodeGemma: Open Code Models Based on Gemma
• LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion
• VersiCode: Towards Version-controllable Code Generation
• A Survey on Large Language Models for Code Generation
• MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark
• LoRA Learns Less and Forgets Less
• Knowledge Conflicts for LLMs: A Survey
• Data Interpreter: An LLM Agent For Data Science
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• A Survey on Large Language Models for Software Engineering
• Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Code Llama: Open Foundation Models for Code
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Editing Large Language Models: Problems, Methods, and Opportunities
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
• Large Language Models Meet NL2Code: A Survey
• DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
• Training language models to follow instructions with human feedback
• Entity-Based Knowledge Conflicts in Question Answering
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• Measuring Mathematical Problem Solving With the MATH Dataset
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Proximal Policy Optimization Algorithms

Cited by

• DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learning
• CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
• MENTOR: A Reinforcement Learning Framework for Enabling Tool Use in Small Models via Teacher-Optimized Rewards
• MoL-RL: Distilling Multi-Step Environmental Feedback into LLMs for Feedback-Independent Reasoning

Related

• CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
• CodeUpdateArena: Benchmarking Knowledge Editing on API Updates
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
• StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
• CodeBoost: Boosting Code LLMs by Squeezing Knowledge from Code Snippets with RL