arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.09467cs.LG
收藏

AQuA:一个用于标签质量评估的基准测试工具

AQuA: A Benchmarking Tool for Label Quality Assessment

Carnegie Mellon University(卡内基梅隆大学)

Mononito Goswami, Vedant Sanil, Arjun Choudhry, Arvind Srinivasan, Chalisa Udompanyawit, Artur Dubrawski

AI总结:

针对标签噪声下机器学习方法缺乏统一评估基准的问题,本文提出AQuA基准测试环境及标签错误检测模型的设计空间,以支持严格、客观地评估和选择标签质量评估工具。

摘要:

机器学习(ML)模型的好坏取决于其训练所用的数据。但近期研究发现,广泛用于训练和评估ML模型的数据集(例如ImageNet)普遍存在标注错误。训练集上的错误标签会损害ML模型的泛化能力,并影响使用测试集进行的评估和模型选择。因此,在存在标注错误的情况下进行学习是一个活跃的研究领域,但该领域缺乏一个全面的基准来评估这些方法。这些方法大多在少数几个计算机视觉数据集上进行评估,且实验协议存在显著差异。由于方法众多且评估不一致,ML从业者如何选择合适的模型来评估其数据中的标签质量也变得不明确。为此,我们提出了一个基准测试环境AQuA,以严格评估在标签噪声存在下支持机器学习的方法。我们还引入了一个设计空间,用以界定标签错误检测模型的具体设计选择。我们希望所提出的设计空间和基准能够帮助从业者选择正确的工具来提升标签质量,并希望我们的基准能够对面对错误标注数据的机器学习工具进行客观且严格的评估。

英文摘要:

Machine learning (ML) models are only as good as the data they are trained on. But recent studies have found datasets widely used to train and evaluate ML models, e.g. ImageNet, to have pervasive labeling errors. Erroneous labels on the train set hurt ML models' ability to generalize, and they impact evaluation and model selection using the test set. Consequently, learning in the presence of labeling errors is an active area of research, yet this field lacks a comprehensive benchmark to evaluate these methods. Most of these methods are evaluated on a few computer vision datasets with significant variance in the experimental protocols. With such a large pool of methods and inconsistent evaluation, it is also unclear how ML practitioners can choose the right models to assess label quality in their data. To this end, we propose a benchmarking environment AQuA to rigorously evaluate methods that enable machine learning in the presence of label noise. We also introduce a design space to delineate concrete design choices of label error detection models. We hope that our proposed design space and benchmark enable practitioners to choose the right tools to improve their label quality and that our benchmark enables objective and rigorous evaluation of machine learning tools facing mislabeled data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑