Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges

2025/06/01 by Muzsai, Lajos, Imolai, David, Lukács, András · 2 citations
#68M25 #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #I.2.1 #K.6.5

paper · doi:10.48550/arxiv.2506.02048

Abstract

We present 'Random-Crypto', a procedurally generated cryptographic Capture The Flag (CTF) dataset designed to unlock the potential of Reinforcement Learning (RL) for LLM-based agents in security-sensitive domains. Cryptographic reasoning offers an ideal RL testbed: it combines precise validation, structured multi-step inference, and reliance on reliable computational tool use. Leveraging these properties, we fine-tune a Python tool-augmented Llama-3.1-8B via Group Relative Policy Optimization (GRPO) in a secure execution environment. The resulting agent achieves a significant improvement in Pass@8 on previously unseen challenges. Moreover, the improvements generalize to two external benchmarks: 'picoCTF', spanning both crypto and non-crypto tasks, and 'AICrypto MCQ', a multiple-choice benchmark of 135 cryptography questions. Ablation studies attribute the gains to enhanced tool usage and procedural reasoning. These findings position 'Random-Crypto' as a rich training ground for building intelligent, adaptable LLM agents capable of handling complex cybersecurity tasks.

Citations

• AICrypto: Evaluating Cryptography Capabilities of Large Language Models
• Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
• NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
• Teams of LLM Agents can Exploit Zero-Day Vulnerabilities
• LLM Agents can Autonomously Exploit One-day Vulnerabilities
• LLM Agents can Autonomously Hack Websites
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions
• PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
• InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
• QLoRA: Efficient Finetuning of Quantized LLMs
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Training language models to follow instructions with human feedback
• Proximal Policy Optimization Algorithms
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

Cited by

• AgentCyTE: Leveraging Agentic AI to Generate Cybersecurity Training & Experimentation Scenarios
• Attacks and Defenses Against LLM Fingerprinting

Related

• AICrypto: Evaluating Cryptography Capabilities of Large Language Models
• Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark
• CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenges
• Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity games
• D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security