SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows

2025/06/08 by Rebecca Saul, Saul, Rebecca, Hao Wang +5 · 1 citation
Computer Science · Decision Sciences · #Advanced Malware Detection Techniques #Adversarial Robustness in Machine Learning #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Scientific Computing and Data Management

paper · pdf · doi:10.48550/arxiv.2506.07313

openalex publication_date 2025/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have seen widespread success in code generation tasks for different scenarios, both everyday and professional. However current LLMs, despite producing functional code, do not prioritize security and may generate code with exploitable vulnerabilities. In this work, we propose techniques for generating code that is more likely to be secure and introduce SCGAgent, a proactive secure coding agent that implements our techniques. We use security coding guidelines that articulate safe programming practices, combined with LLM-generated unit tests to preserve functional correctness. In our evaluation, we find that SCGAgent is able to preserve nearly 98% of the functionality of the base Sonnet-3.7 LLM while achieving an approximately 25% improvement in security. Moreover, SCGAgent is able to match or best the performance of sophisticated reasoning LLMs using a non-reasoning model and an agentic workflow.

Citations

• Trading Inference-Time Compute for Adversarial Robustness
• s1: Simple test-time scaling
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
• CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings
• SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI
• AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing
• The Llama 3 Herd of Models
• Prompting Techniques for Secure Code Generation: A Systematic Investigation
• R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• Constrained Decoding for Secure Code Generation
• Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineering
• Instruction Tuning for Secure Code Generation
• LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks
• Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
• Magicoder: Empowering Code Generation with OSS-Instruct
• Assessing the Security of GitHub Copilot Generated Code -- A Targeted Replication Study
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Code Llama: Open Foundation Models for Code
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct
• StarCoder: may the source be with you!
• ChatUniTest: A Framework for LLM-Based Test Generation
• No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
• Toolformer: Language Models Can Teach Themselves to Use Tools
• The Stack: 3 TB of permissively licensed source code
• InCoder: A Generative Model for Code Infilling and Synthesis
• Training Compute-Optimal Large Language Models
• Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
• Evaluating Large Language Models Trained on Code
• Scaling Laws for Neural Language Models
• OpenAI o1 System Card
• ASTER: Natural and Multi-language Unit Test Generation with LLMs

Cited by

• PurpCode: Reasoning for Safer Code Generation

Related

• SecureAgentBench: Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios
• Fortifying LLM-Based Code Generation with Graph-Based Reasoning on Secure Coding Practices
• Ocassionally Secure: A Comparative Analysis of Code Generation Assistants
• A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
• TypePilot: Leveraging the Scala Type System for Secure LLM-generated Code