谁在NLP中进行标注？2018年至2025年人工标注报告的大规模评估
Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
AI总结：
本研究通过大规模审计NLP论文中的人工标注报告，发现标注细节报告不完整，并提出了改进报告质量的框架和建议。
摘要：
人工标注是许多NLP研究的经验基础，从数据集构建到模型评估，但论文往往不清楚谁产生了标注以及如何控制标注过程。我们首次对主要NLP会议中的人工标注报告进行了大规模、任务级别的审计，询问哪些标注细节被记录，哪些缺失，以及报告如何随时间、主题、会议和人工判断的预期用途而变化。我们引入了一个统一的标注报告实践分类法，并针对Annotated-gold（一个由41篇论文和72个标注任务组成的人工裁决黄金标准）验证了一个LLM辅助的提取流程，其中最佳模型与裁决标签达到了与人类相当的一致性，Krippendorff's alpha为0.606，而人类间一致性为0.585。利用该流程，我们构建了Annotated-llm数据集，涵盖2018-2025年ACL会议论文，包含来自1603篇论文的2667个提取的标注任务，发现论文经常报告操作细节，如招募策略、标注者专业知识和标注量，但往往省略评估标注有效性所需的细节，包括培训、语言能力、报酬、社会人口统计、裁决和一致性值，尤其是在模型评估研究中。我们的结果表明，NLP中的标注报告随时间有所改善，但仍不均衡，我们建立了一个可扩展的框架和最低报告建议，以使人工标注更可靠、可重复和可解释。
英文摘要：
Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled. We provide the first large-scale, task-level audit of human annotation reporting across major NLP venues, asking which annotation details are documented, which are missing, and how reporting varies across time, topic, venue, and intended use of human judgment. We introduce a unified taxonomy of annotation-reporting practices and validate an LLM-assisted extraction pipeline against Annotated-gold, a human-adjudicated gold standard of 41 papers and 72 annotation tasks, where the best model reaches human-comparable agreement with adjudicated labels, with Krippendorff's alpha of 0.606 versus 0.585 for human-human agreement. Using this pipeline, we construct Annotated-llm, a dataset covering ACL-venue papers from 2018-2025, with 2,667 extracted annotation tasks from 1,603 papers, and find that papers frequently report operational details such as recruitment strategies, annotator expertise, and annotation volume, but often omit details needed to assess annotation validity, including training, language proficiency, compensation, socio-demographics, adjudication, and agreement values, especially in model-evaluation studies. Our results show that annotation reporting in NLP has improved over time but remains uneven, and they establish a scalable framework and bare-minimum reporting recommendations for making human annotation more reliable, reproducible, and interpretable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
