2020/12/16 by Sean Oesch, Oesch, Sean, Robert A. Bridges +13 · 4 citations
Computer Science · #Information and Cyber Security #Software Engineering Research #Software System Performance and Reliability
paper · pdf · doi:10.48550/arxiv.2012.09013
Gartner, a large research and advisory company, anticipates that by 2024 80%\nof security operation centers (SOCs) will use machine learning (ML) based\nsolutions to enhance their operations. In light of such widespread adoption, it\nis vital for the research community to identify and address usability concerns.\nThis work presents the results of the first in situ usability assessment of\nML-based tools. With the support of the US Navy, we leveraged the national\ncyber range, a large, air-gapped cyber testbed equipped with state-of-the-art\nnetwork and user emulation capabilities, to study six US Naval SOC analysts'\nusage of two tools. Our analysis identified several serious usability issues,\nincluding multiple violations of established usability heuristics form user\ninterface design. We also discovered that analysts lacked a clear mental model\nof how these tools generate scores, resulting in mistrust and/or misuse of the\ntools themselves. Surprisingly, we found no correlation between analysts' level\nof education or years of experience and their performance with either tool,\nsuggesting that other factors such as prior background knowledge or personality\nplay a significant role in ML-based tool usage. Our findings demonstrate that\nML-based security tool vendors must put a renewed focus on working with\nanalysts, both experienced and inexperienced, to ensure that their systems are\nusable and useful in real-world security operations settings.\n