Anomaly Detection for Network Connection Logs

2018/12/01 by Mehta, Swapneel, Kothuri, Prasanth, Garcia, Daniel Lanza
#Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.1812.01941

Abstract

We leverage a streaming architecture based on ELK, Spark and Hadoop in order to collect, store, and analyse database connection logs in near real-time. The proposed system investigates outliers using unsupervised learning; widely adopted clustering and classification algorithms for log data, highlighting the subtle variances in each model by visualisation of outliers. Arriving at a novel solution to evaluate untagged, unfiltered connection logs, we propose an approach that can be extrapolated to a generalised system of analysing connection logs across a large infrastructure comprising thousands of individual nodes and generating hundreds of lines in logs per second.

Related

• Real-Time Outlier Connections Detection in Databases Network Traffic
• Log Analysis Case Study Using LoGS
• Distributed Anomaly Detection in Edge Streams using Frequency based Sketch Datastructures
• Deep Learning-based Anomaly Detection and Log Analysis for Computer Networks
• Graph Neural Network and Transformer Integration for Unsupervised System Anomaly Discovery