arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2508.04489cs.LGcs.AI
收藏

层次评分用于机器学习分类器误差影响评估

Hierarchical Scoring for Machine Learning Classifier Error Impact Evaluation

National Security Institute, Virginia Tech(维吉尼亚理工大学国家安全研究所);Applied Research Corporation, Virginia Tech(维吉尼亚理工大学应用研究公司)

Erin Lanus, Daniel Wolodkin, Laura J. Freeman

AI总结:

本文提出层次评分度量,通过评分树评估分类器误差影响,实现更细粒度的模型性能评估。

摘要:

机器学习(ML)模型的一个常见用途是预测样本的类别。目标检测是分类的扩展,它通过在样本内使用边界框来定位对象。分类,以及由此扩展的目标检测,通常通过将预测标签与地面真实标签不匹配来判定预测错误。这种通过/失败评分将所有误分类视为等同。在许多情况下,类标签可以组织成具有层次结构的类学分类,以反映数据之间的关系或操作员对误分类的估值。当存在这样的层次结构时,层次评分度量可以返回给定预测的模型性能,与预测和地面真实标签之间的距离有关。这些度量可以被视为对预测给予部分信用,而不是通过/失败,从而实现更细致的理解误分类的影响。本文开发了不同复杂度的层次评分度量,利用评分树来编码类标签之间的关系,产生反映评分树中距离的度量。这些评分度量在抽象使用案例中得到了演示,其中评分树代表三种加权策略,并通过被禁止的错误类型进行评估。结果表明,这些度量能够更细致地捕捉错误,评分树使调参成为可能。本文展示了一种评估ML性能的方法,该方法不仅根据所犯错误的数量对模型进行排名,还根据错误的类型或影响进行排名。评分度量的Python实现将在发表时通过开源存储库提供。

英文摘要:

A common use of machine learning (ML) models is predicting the class of a sample. Object detection is an extension of classification that includes localization of the object via a bounding box within the sample. Classification, and by extension object detection, is typically evaluated by counting a prediction as incorrect if the predicted label does not match the ground truth label. This pass/fail scoring treats all misclassifications as equivalent. In many cases, class labels can be organized into a class taxonomy with a hierarchical structure to either reflect relationships among the data or operator valuation of misclassifications. When such a hierarchical structure exists, hierarchical scoring metrics can return the model performance of a given prediction related to the distance between the prediction and the ground truth label. Such metrics can be viewed as giving partial credit to predictions instead of pass/fail, enabling a finer-grained understanding of the impact of misclassifications. This work develops hierarchical scoring metrics varying in complexity that utilize scoring trees to encode relationships between class labels and produce metrics that reflect distance in the scoring tree. The scoring metrics are demonstrated on an abstract use case with scoring trees that represent three weighting strategies and evaluated by the kind of errors discouraged. Results demonstrate that these metrics capture errors with finer granularity and the scoring trees enable tuning. This work demonstrates an approach to evaluating ML performance that ranks models not only by how many errors are made but by the kind or impact of errors. Python implementations of the scoring metrics will be available in an open-source repository at time of publication.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑