PIPer: On-Device Environment Setup via Online Reinforcement Learning

2025/09/29 by Alexander Kovrigin, Aleksandra Eliseeva, Kovrigin, Alexander +7
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Green IT and Sustainability #IoT and Edge/Fog Computing #Machine Learning (cs.LG) #Modular Robots and Swarm Intelligence #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2509.25455

openalex publication_date 2025/09/29 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Environment setup-the process of configuring the system to work with a specific software project-represents a persistent challenge in Software Engineering (SE). Automated environment setup methods could assist developers by providing fully configured environments for arbitrary repositories without manual effort. This also helps SE researchers to scale execution-based benchmarks. However, recent studies reveal that even state-of-the-art Large Language Models (LLMs) achieve limited success in automating this task. To address this limitation, we tune a specialized model for environment setup. We combine supervised fine-tuning for generating correct Bash scripts and Reinforcement Learning with Verifiable Rewards (RLVR) to adapt it to the task of environment setup. On EnvBench-Python, our method enables Qwen3-8B (a model runnable on consumer hardware) to perform on par with larger models-Qwen3-32B and GPT-4o. The training code and model checkpoints are available online: https://github.com/JetBrains-Research/PIPer.

Citations

• RL's Razor: Why Online Reinforcement Learning Forgets Less
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning
• Tool-integrated Reinforcement Learning for Repo Deep Search
• SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments
• Assessing Small Language Models for Code Generation: An Empirical Study with Benchmarks
• Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs
• AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
• SWE-bench Goes Live!
• Reinforcing General Reasoning without Verifiers
• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
• SweRank: Software Issue Localization with Code Ranking
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data
• SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
• SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
• Understanding R1-Zero-Like Training: A Critical Perspective
• Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
• EnvBench: A Benchmark for Automated Environment Setup
• LocAgent: Graph-Guided LLM Agents for Code Localization
• Automated Benchmark Generation for Repository-Level Coding Tasks
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Multi-Turn Code Generation Through Single-Step Rewards
• Bridging Bug Localization and Issue Fixing: A Hierarchical Localization Framework Leveraging Large Language Models
• Repo2Run: Automated Building Executable Environment for Code Repository at Scale
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Training Software Engineering Agents and Verifiers with SWE-Gym
• Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects
• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
• Large Language Model-Based Agents for Software Engineering: A Survey
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• A Survey of Reinforcement Learning from Human Feedback
• Persistent memory for AI coding agents: a pre-registered SWE-bench Verified benchmark
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• Distilling the Knowledge in a Neural Network
• SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks
• A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning

Related

• EnvBench: A Benchmark for Automated Environment Setup
• EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine
• PufferLib: Making Reinforcement Learning Libraries and Environments Play Nice
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
• SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development