arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.13801cs.LGcs.AI
收藏

通过多级标注者建模提高评估的可重复性

Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling

Rochester Institute of Technology(罗切斯特理工学院);Google Research(谷歌研究)

Deepak Pandita, Flip Korn, Chris Welty, Christopher M. Homan

AI总结:

本文提出多级抽样方法,通过分析标注者数量与响应数量的平衡,提升评估结果的可重复性与统计显著性。

摘要:

随着生成式AI模型如大语言模型(LLMs)的普及,确保其安全性、鲁棒性和整体可信度至关重要。然而,AI正面临由不可靠评估和不可重复实验结果驱动的可重复性危机。尽管人类评估者常用于评估模型的效用和安全性,但他们的注释引入了异质偏见和主观意见。克服这种变异性极具挑战性,因为很少有数据能研究实验可重复性如何随着标注者池的增长而提高。标准评估实践通常依赖每个项目少量注释(通常为3到5个),且缺乏必要的持久标注者标识符来建模跨项目的个体变异性。在本文中,我们引入了多级抽样方法,以真实地建模标注者行为。利用具有大量评分和持久标注者标识符的数据集,我们分析了为实现统计显著性所需的项目数(N)和每个项目响应数(K)之间的权衡。

英文摘要:

As generative AI models such as large language models (LLMs) become more pervasive, ensuring the safety, robustness, and overall trustworthiness of these systems is paramount. However, AI is currently facing a reproducibility crisis driven by unreliable evaluations and unrepeatable experimental results. While human raters are often used to assess models for utility and safety, they introduce divergent biases and subjective opinions into their annotations. Overcoming this variance is exceptionally challenging because very little data exists to study how experimental repeatability actually improves as the annotator pool grows. Standard evaluation practices typically rely on a small number of annotations per item (often 3 to 5) and lack the persistent rater identifiers necessary to model individual variance across items. In this work, we introduce a multi-level bootstrapping approach to model annotator behavior realistically. Leveraging datasets with a large number of ratings and persistent rater identifiers, we analyze the tradeoffs between the number of items ($N$) and the number of responses per item ($K$) required to achieve statistical significance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑