Donkii：标注错误检测方法能否发现指令微调数据集中的错误？
Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-Tuning Datasets?
AI总结：
本文提出首个面向指令微调数据的标注错误检测基准DONKII，构建错误分类法并评估四种生成式AED基线，证明方法选择与模型规模对清洗指令微调数据至关重要。
摘要：
指令微调已成为大语言模型（LLMs）训练流程中不可或缺的一部分，并已被证明能带来显著的性能提升。在一条正交的研究路线中，标注错误检测（AED）已成为一种用于检测金标准标签中质量问题的工具。然而，迄今为止，AED方法的应用仅限于分类任务。AED方法在语言生成场景中的泛化能力如何，仍是一个悬而未决的问题，而这类场景正通过LLMs变得越来越普遍。在本文中，我们提出了首个新颖的针对指令微调数据的AED基准：DONKII。该基准包含三个指令微调数据集，并通过专家和半自动方法对其进行了错误标注的增强。我们还为指令微调数据提供了一种新的错误类型分类法。我们发现，所有三个数据集都包含明显的错误，这些错误有时会直接传播到经过指令微调的LLMs中。我们针对生成式场景提出了四种AED基线方法，并在新引入的数据集上对它们进行了广泛评估。我们的结果表明，选择正确的AED方法和模型规模确实至关重要，并由此得出了关于如何使用AED方法清洗指令微调数据的实用建议。
英文摘要：
Instruction tuning has become an integral part of training pipelines for Large Language Models (LLMs) and has been shown to yield strong performance gains. In an orthogonal line of research, Annotation Error Detection (AED) has emerged as a tool for detecting quality problems in gold standard labels. So far, however, the application of AED methods has been limited to classification tasks. It is an open question how well AED methods generalize to language generation settings, which are becoming more widespread via LLMs. In this paper, we present a first and novel benchmark for AED on instruction tuning data: DONKII. It comprises three instruction-tuning datasets enriched with error annotations by experts and semi-automatic methods. We also provide a novel taxonomy of error types for instruction-tuning data. We find that all three datasets contain clear errors, which sometimes propagate directly into instruction-tuned LLMs. We propose four AED baselines for the generative setting and evaluate them extensively on the newly introduced dataset. Our results show that the choice of the right AED method and model size is indeed crucial and derive practical recommendations for how to use AED methods to clean instruction-tuning data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
