arXiv 2012.090132020-12-16cs.HCcs.CRcs.LG

安全运营中心基于机器学习的工具的可用性评估

An Assessment of the Usability of Machine Learning Based Tools for the Security Operations Center

Oak Ridge National Laboratory（橡树岭国家实验室）；Assured Information Security（可信信息安全公司）

Sean Oesch, Robert Bridges, Jared Smith, Justin Beaver, John Goodall, Kelly Huffer, Craig Miles, Dan Scofield

摘要：

大型研究和咨询公司Gartner预计，到2024年，80%的安全运营中心（SOC）将使用基于机器学习（ML）的解决方案来增强其运营。鉴于这种广泛的采用，研究社区识别和解决可用性问题至关重要。这项工作提出了第一次在现场可用性评估的ML为基础的工具的结果。在美国海军的支持下，我们利用国家网络靶场，一个配备最先进网络和用户仿真功能的大型气隙网络测试平台，研究了六名美国海军SOC分析师对两种工具的使用情况。我们的分析确定了几个严重的可用性问题，包括多个违反既定可用性原则的用户界面设计。我们还发现，分析师对这些工具如何生成分数缺乏清晰的心理模型，导致对工具本身的不信任和/或滥用。令人惊讶的是，我们发现分析师的教育水平或多年的经验和他们的表现与任何一个工具之间没有相关性，这表明其他因素，如以前的背景知识或个性发挥了显着的作用，在ML为基础的工具的使用。我们的研究结果表明，基于ML的安全工具供应商必须重新关注与经验丰富和缺乏经验的分析师的合作，以确保他们的系统在现实世界的安全操作设置中可用和有用。

英文摘要：

Gartner, a large research and advisory company, anticipates that by 2024 80% of security operation centers (SOCs) will use machine learning (ML) based solutions to enhance their operations. In light of such widespread adoption, it is vital for the research community to identify and address usability concerns. This work presents the results of the first in situ usability assessment of ML-based tools. With the support of the US Navy, we leveraged the national cyber range, a large, air-gapped cyber testbed equipped with state-of-the-art network and user emulation capabilities, to study six US Naval SOC analysts' usage of two tools. Our analysis identified several serious usability issues, including multiple violations of established usability heuristics form user interface design. We also discovered that analysts lacked a clear mental model of how these tools generate scores, resulting in mistrust and/or misuse of the tools themselves. Surprisingly, we found no correlation between analysts' level of education or years of experience and their performance with either tool, suggesting that other factors such as prior background knowledge or personality play a significant role in ML-based tool usage. Our findings demonstrate that ML-based security tool vendors must put a renewed focus on working with analysts, both experienced and inexperienced, to ensure that their systems are usable and useful in real-world security operations settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文