Data Drift Monitoring for Log Anomaly Detection Pipelines

2023/10/17 by Dipak Wani, Samuel Ackerman, Wani, Dipak +9
Computer Science · #Anomaly Detection Techniques and Applications #Applications (stat.AP) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Network Security and Intrusion Detection #Software System Performance and Reliability #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2310.14893

openalex publication_date 2023/10/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Logs enable the monitoring of infrastructure status and the performance of associated applications. Logs are also invaluable for diagnosing the root causes of any problems that may arise. Log Anomaly Detection (LAD) pipelines automate the detection of anomalies in logs, providing assistance to site reliability engineers (SREs) in system diagnosis. Log patterns change over time, necessitating updates to the LAD model defining the `normal' log activity profile. In this paper, we introduce a Bayes Factor-based drift detection method that identifies when intervention, retraining, and updating of the LAD model are required with human involvement. We illustrate our method using sequences of log activity, both from unaltered data, and simulated activity with controlled levels of anomaly contamination, based on real collected log data.

Related

• LogSD: Detecting Anomalies from System Logs through Self-supervised Learning and Frequency-based Masking
• Log-based Anomaly Detection of Enterprise Software: An Empirical Study
• What Information Contributes to Log-based Anomaly Detection? Insights from a Configurable Transformer-Based Approach
• LogDP: Combining Dependency and Proximity for Log-based Anomaly Detection
• Unsupervised Log Anomaly Detection with Few Unique Tokens