Progressive Agent Skill Generation via Reinforcement Learning

2026/08/03 by Junhao Shen, Zhanqiu Zhang, Yiwen Guo +1
Computer Science · #cs.LG #cs.CL

paper · pdf

Code is available at https://github.com/ejhshen/skill-alpha

arxiv created 2026/08/03 · arxiv updated 2026/08/04

Abstract

Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the agent on downstream tasks. To address this challenge, we propose Skill-α, a reinforcement learning method for progressively generating high-quality agent skills. Specifically, we formulate skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits, and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill-α generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill-α improves average downstream success rates over the strongest skill-generation baseline by 3.3 points on CL-Bench and 6.7 points on tau2-bench. Further ablations validate the importance of rollback reward and progressive generation.

Citations

• SkillOS: Learning Skill Curation for Self-Evolving Agents
• From Context to Skills: Can Language Models Learn from Context Skillfully?
• SkillRouter: Skill Routing for LLM Agents at Scale
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
• Reinforcement Learning for Self-Improving Agent with Skill Library
• Achieving Olympia-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learning
• Mem-α: Learning Memory Construction via Reinforcement Learning
• Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
• Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
• τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
• Qwen3 Technical Report
• Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
• SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation
• ToRL: Scaling Tool-Integrated RL
• Large Language Model Agent: A Survey on Methodology, Applications and Challenges
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
• LLM With Tools: A Survey
• Agent Workflow Memory
• Multi-Step Reasoning with Large Language Models, a Survey
• SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation
• Understanding the planning of LLM agents: A survey
• ExpeL: LLM Agents Are Experiential Learners
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Proximal Policy Optimization Algorithms