Anomaly Detection in Large Scale Networks with Latent Space Models

2019/11/13 by Lee, Wesley, McCormick, Tyler H., Neil, Joshua +2 · 1 citation
#Applications (stat.AP) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (stat.ML) #Methodology (stat.ME) #Social and Information Networks (cs.SI)

paper · doi:10.48550/arxiv.1911.05522

Abstract

We develop a real-time anomaly detection algorithm for directed activity on large, sparse networks. We model the propensity for future activity using a dynamic logistic model with interaction terms for sender- and receiver-specific latent factors in addition to sender- and receiver-specific popularity scores; deviations from this underlying model constitute potential anomalies. Latent nodal attributes are estimated via a variational Bayesian approach and may change over time, representing natural shifts in network activity. Estimation is augmented with a case-control approximation to take advantage of the sparsity of the network and reduces computational complexity from O(N²) to O(E), where N is the number of nodes and E is the number of observed edges. We run our algorithm on network event records collected from an enterprise network of over 25,000 computers and are able to identify a red team attack with half the detection rate required of the model without latent interaction terms.

Cited by

• On fair and realistic performance evaluations for graph-based lateral movement detectors

Related

• Anomaly detection in the dynamics of web and social networks
• Anomaly Detection in Dynamic Networks of Varying Size
• Size-Consistent Statistics for Anomaly Detection in Dynamic Networks
• Low Latency Anomaly Detection and Bayesian Network Prediction of Anomaly Likelihood
• Community detection and anomaly prediction in dynamic networks