Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice

2024/03/26 by Jake Hesford, Daniel Cheng, Hesford, Jake +11 · 2 citations
Computer Science · #68M20 #68M25 #C.4 #Cryptography and Security (cs.CR) #D.m #FOS: Computer and information sciences #Machine Learning (cs.LG) #Network Security and Intrusion Detection

paper · pdf · doi:10.48550/arxiv.2403.17458

openalex publication_date 2024/03/26 · openalex created_date 2024/03/30 · openalex updated_date 2026/07/28

Abstract

Our paper provides empirical comparisons between recent IDSs to provide an objective comparison between them to help users choose the most appropriate solution based on their requirements. Our results show that no one solution is the best, but is dependent on external variables such as the types of attacks, complexity, and network environment in the dataset. For example, BoT_IoT and Stratosphere IoT datasets both capture IoT-related attacks, but the deep neural network performed the best when tested using the BoT_IoT dataset while HELAD performed the best when tested using the Stratosphere IoT dataset. So although we found that a deep neural network solution had the highest average F1 scores on tested datasets, it is not always the best-performing one. We further discuss difficulties in using IDS from literature and project repositories, which complicated drawing definitive conclusions regarding IDS selection.

Cited by

• StealthCup: Realistic, Multi-Stage, Evasion-Focused CTF for Benchmarking IDS
• An Evaluation Framework for Network IDS/IPS Datasets: Leveraging MITRE ATT&CK and Industry Relevance Metrics

Related

• Emerging Trends of Recently Published Datasets for Intrusion Detection Systems (IDS): A Survey
• A Robust Comparison of the KDDCup99 and NSL-KDD IoT Network Intrusion Detection Datasets Through Various Machine Learning Algorithms
• Shallow and Deep Networks Intrusion Detection System: A Taxonomy and Survey
• Optimized IoT Intrusion Detection using Machine Learning Technique
• An Evaluation Framework for Network IDS/IPS Datasets: Leveraging MITRE ATT&CK and Industry Relevance Metrics