Dynamic Risk Assessments for Offensive Cybersecurity Agents

2025/05/23 by Boyi Wei, Wei, Boyi, Benedikt Stroebl +9 · 1 voice · 3 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Information and Cyber Security #Network Security and Intrusion Detection #Smart Grid Security and Resilience #cs.AI #cs.CR

paper · pdf · doi:10.48550/arxiv.2505.18384

openalex publication_date 2025/05/23 · arxiv published 2025/05/23 · openalex created_date 2025/10/10 · arxiv updated 2025/10/30 · openalex updated_date 2026/07/28

Abstract

Foundation models are increasingly becoming better autonomous programmers, raising the prospect that they could also automate dangerous offensive cyber-operations. Current frontier model audits probe the cybersecurity risks of such agents, but most fail to account for the degrees of freedom available to adversaries in the real world. In particular, with strong verifiers and financial incentives, agents for offensive cybersecurity are amenable to iterative improvement by would-be adversaries. We argue that assessments should take into account an expanded threat model in the context of cybersecurity, emphasizing the varying degrees of freedom that an adversary may possess in stateful and non-stateful environments within a fixed compute budget. We show that even with a relatively small compute budget (8 H100 GPU Hours in our study), adversaries can improve an agent's cybersecurity capability on InterCode CTF by more than 40% relative to the baseline -- without any external assistance. These results highlight the need to evaluate agents' cybersecurity risk in a dynamic manner, painting a more representative picture of risk.

Citations

• Absolute Zero: Reinforced Self-play Reasoning with Zero Data
• Toward Efficient Exploration by Large Language Model Agents
• Gemma 3 Technical Report
• A Framework for Evaluating Emerging Cyberattack Capabilities of AI
• AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defenses
• Forecasting Frontier Language Model Agent Capabilities
• OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilities
• On Evaluating the Durability of Safeguards for Open-Weight LLMs
• Hacking CTFs with Plain Agents
• One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity
• An Adversarial Perspective on Machine Unlearning for AI Safety
• Training Language Models to Self-Correct via Reinforcement Learning
• Qwen2.5-Coder Technical Report
• Planning In Natural Language Improves LLM Search For Code Generation
• Generative Verifiers: Reward Modeling as Next-Token Prediction
• Automated Design of Agentic Systems
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Recursive Introspection: Teaching Language Model Agents How to Self-Improve
• Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design
• NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
• Teams of LLM Agents can Exploit Zero-Day Vulnerabilities
• CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models
• LLM Agents can Autonomously Exploit One-day Vulnerabilities
• The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks
• On the Societal Impact of Open Foundation Models
• An Empirical Evaluation of LLMs for Solving Offensive Security Challenges
• CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge
• V-STaR: Training Verifiers for Self-Taught Reasoners
• LLM Agents can Autonomously Hack Websites
• SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security
• ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
• Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
• PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
• Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions
• PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
• Getting pwn’d by AI: Penetration Testing with Large Language Models
• InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Automatic Prompt Optimization with "Gradient Descent" and Beam Search
• Self-Refine: Iterative Refinement with Self-Feedback
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Aligning Language Models with Preferences through f-divergence Minimization
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Large Language Models Can Self-Improve
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• STaR: Bootstrapping Reasoning With Reasoning
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• WebGPT: Browser-assisted question-answering with human feedback
• Evaluating Large Language Models Trained on Code
• Learning to summarize from human feedback
• The Limits of Inference Scaling Through Resampling

Cited by

• Large Language Models Develop Novel Social Biases Through Adaptive Exploration
• PoCo: Agentic Proof-of-Concept Exploit Generation for Smart Contracts
• Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Models

Discussions

• Evaluations of offensive cyber abilities of frontier models often assume a fixed, 1-shot system. But this is just the tip of the iceberg, the risk "bubble" is bigger (and cheaper 💰) than you think. E [bsky, 3 points, 1 comments]

Related

• Toward Quantitative Modeling of Cybersecurity Risks Due to AI Misuse
• Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs
• Frontier AI's Impact on the Cybersecurity Landscape
• OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilities
• Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response