IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection

2025/12/16 by Roman Nekrasov, Stefano Fossati, Nekrasov, Roman +7 · 1 voice · 1 citation
Computer Science · Engineering · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #BIM and Construction Integration #Benchmark (surveying) #Cloud computing #Code generation #Emulation #FOS: Computer and information sciences #Graph #Knowledge graph #Multimodal Machine Learning Applications #Software Engineering (cs.SE) #Taxonomy (biology) #cs.AI #cs.SE

paper · pdf · open access · doi:10.48550/arxiv.2512.14792

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/12/16 · arxiv published 2025/12/16 · arxiv updated 2025/12/16 · openalex created_date 2025/12/19 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) currently exhibit low success rates in generating correct and intent-aligned Infrastructure as Code (IaC). This research investigated methods to improve LLM-based IaC generation, specifically for Terraform, by systematically injecting structured configuration knowledge. To facilitate this, an existing IaC-Eval benchmark was significantly enhanced with cloud emulation and automated error analysis. Additionally, a novel error taxonomy for LLM-assisted IaC code generation was developed. A series of knowledge injection techniques was implemented and evaluated, progressing from Naive Retrieval-Augmented Generation (RAG) to more sophisticated Graph RAG approaches. These included semantic enrichment of graph components and modeling inter-resource dependencies. Experimental results demonstrated that while baseline LLM performance was poor (27.1% overall success), injecting structured configuration knowledge increased technical validation success to 75.3% and overall success to 62.6%. Despite these gains in technical correctness, intent alignment plateaued, revealing a "Correctness-Congruence Gap" where LLMs can become proficient "coders" but remain limited "architects" in fulfilling nuanced user intent.

Citations

• Cloud Infrastructure Management in the Age of AI Agents
• Knowledge Graph-Guided Retrieval Augmented Generation
• Advanced RAG Models with Graph Structures: Optimizing Complex Knowledge Reasoning and Text Generation
• Graph Retrieval-Augmented Generation: A Survey
• Large Language Model Enhanced Knowledge Representation Learning: A Survey
• DocCGen: Document-based Controlled Code Generation
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization
• A Survey of using Large Language Models for Generating Infrastructure as Code
• Bugs in Large Language Models Generated Code: An Empirical Study
• G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
• Insights into Natural Language Database Query Errors: From Attention Misalignment to User Handling Strategies
• CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Enhancing Text-based Knowledge Graph Completion with Zero-Shot Large Language Models: A Focus on Semantic Enhancement
• Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation
• Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
• Large Language Models Meet NL2Code: A Survey
• Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering
• Training language models to follow instructions with human feedback
• KGSecConfig: A Knowledge Graph Based Approach for Secured Container Orchestrator Configuration
• Improving language models by retrieving from trillions of tokens
• Evaluating Large Language Models Trained on Code
• Language Models are Few-Shot Learners
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Dense Passage Retrieval for Open-Domain Question Answering
• Knowledge Graphs
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Attention Is All You Need
• Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
• Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models

Cited by

• Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation

Discussions

• Link to the study "IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection" arxiv.org/abs/2512.14792 [bsky, 1 points, 0 comments]

Related

• A Survey of using Large Language Models for Generating Infrastructure as Code
• Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats
• Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation
• SWE-InfraBench: Evaluating Language Models on Cloud Infrastructure Code
• Deployability-Centric Infrastructure-as-Code Generation: An LLM-based Iterative Framework