LogSage: An LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation

2025/06/04 by Weiya Xu, Xu, Weiyuan, Luo, Juntao +16 · 1 citation
Computer Science · Engineering · #Advanced Data Storage Technologies #FOS: Computer and information sciences #Industrial Vision Systems and Defect Detection #Mineral Processing and Grinding #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2506.03691

openalex publication_date 2025/06/04 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Continuous Integration and Deployment (CI/CD) pipelines are critical to modern software engineering, yet diagnosing and resolving their failures remains complex and labor-intensive. We present LogSage, the first end-to-end LLM-powered framework for root cause analysis (RCA) and automated remediation of CI/CD failures. LogSage employs a token-efficient log preprocessing pipeline to filter noise and extract critical errors, then performs structured diagnostic prompting for accurate RCA. For solution generation, it leverages retrieval-augmented generation (RAG) to reuse historical fixes and invokes automation fixes via LLM tool-calling. On a newly curated benchmark of 367 GitHub CI/CD failures, LogSage achieves over 98% precision, near-perfect recall, and an F1 improvement of more than 38% points in the RCA stage, compared with recent LLM-based baselines. In a year-long industrial deployment at ByteDance, it processed over 1.07M executions, with end-to-end precision exceeding 80%. These results demonstrate that LogSage provides a scalable and practical solution for automating CI/CD failure management in real-world DevOps workflows.

Citations

• BitsAI-CR: Automated Code Review via LLM in Practice
• LogLLM: Log-based Anomaly Detection Using Large Language Models
• Developing a Llama-Based Chatbot for CI/CD Question Answering: A Case Study at Ericsson
• Systematic Literature Review on Application of Learning-based Approaches in Continuous Integration
• A Survey of AIOps for Failure Management in the Era of Large Language Models
• LLM meets ML: Data-efficient Anomaly Detection on Unstable Logs
• Exploring LLM-based Agents for Root Cause Analysis
• X-lifecycle Learning for Cloud Incident Management using LLMs
• Automated Unit Test Improvement using Large Language Models at Meta
• Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4
• Rewriting the Code: A Simple Method for Large Language Model Augmented Code Search
• LogGPT: Exploring ChatGPT for Log-Based Anomaly Detection
• Large Language Models for Software Engineering: A Systematic Literature Review
• Large Language Models for Software Engineering: A Systematic Literature Review
• Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies
• Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
• Log-based Anomaly Detection Without Log Parsing

Cited by

• Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents

Related

• Leveraging Resolved Incident History for LLM-Assisted Software Bug Diagnosis
• LogRCA: Log-based Root Cause Analysis for Distributed Services
• Auto-repair without test cases: How LLMs fix compilation errors in large industrial embedded code
• BugSwarm: Mining and Continuously Growing a Dataset of Reproducible\n Failures and Fixes
• Exploring LLM-based Agents for Root Cause Analysis