F1分数对网络安全模型是否次优?引入C_score,一种成本感知的模型评估替代方案
Is $F_1$ Score Suboptimal for Cybersecurity Models? Introducing $C_{score}$, a Cost-Aware Alternative for Model Assessment
AI总结:
本文提出成本感知指标C_score以替代F1分数,通过引入假阳性和假阴性成本比率优化模型评估与阈值选择,在五个网络安全数据集上实现平均49%的成本节省。
摘要:
与机器学习分类器相关的错误成本,即假阳性和假阴性,并不相等,且依赖于具体应用。例如,在网络安全应用中,未能检测到攻击的成本与将良性活动标记为攻击的成本截然不同。机器学习模型构建过程中的各种设计选择,如超参数调优和模型选择,使数据科学家能够在这两类错误之间进行权衡。然而,大多数常用的模型质量评估指标,例如基于模型精确率和召回率定义的F1分数,对这两类错误一视同仁,这使得用户难以针对这些错误的实际成本进行优化。在本文中,我们提出了一种新的成本感知指标C_score,它基于精确率和召回率,可以替代F1分数用于模型评估和选择。该指标包含一个成本比率,用于考虑处理假阳性和假阴性的不同成本。我们推导并刻画了这一新成本指标的特性,并将其与F1分数进行了比较。此外,我们将该指标用于五个网络安全相关数据集在多个成本比率下的模型阈值设定。结果表明平均成本节省了49%。
英文摘要:
The cost of errors related to machine learning classifiers, namely, false positives and false negatives, are not equal and are application dependent. For example, in cybersecurity applications, the cost of not detecting an attack is very different from marking a benign activity as an attack. Various design choices during machine learning model building, such as hyperparameter tuning and model selection, allow a data scientist to trade-off between these two errors. However, most of the commonly used metrics to evaluate model quality, such as $F_1$ score, which is defined in terms of model precision and recall, treat both these errors equally, making it difficult for users to optimize for the actual cost of these errors. In this paper, we propose a new cost-aware metric, $C_{score}$ based on precision and recall that can replace $F_1$ score for model evaluation and selection. It includes a cost ratio that takes into account the differing costs of handling false positives and false negatives. We derive and characterize the new cost metric, and compare it to $F_1$ score. Further, we use this metric for model thresholding for five cybersecurity related datasets for multiple cost ratios. The results show an average cost savings of 49%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。