arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.08243cs.LGmath.STstat.TH
收藏

DiPietro-Hazari Kappa:一种通过标注评估标签质量的新指标

DiPietro-Hazari Kappa: A Novel Metric for Assessing Labeling Quality via Annotation

Daniel M. DiPietro, Vivek Hazari

摘要:

数据是现代机器学习的关键组成部分,但用于评估数据标签质量的统计数据在文献中仍然很少。在这里,我们介绍了DiPietro-Hazari Kappa,这是一种新的统计指标,用于评估人类注释背景下建议的数据集标签的质量。DiPietro-Hazari Kappa基于经典的Fleiss Kappa注释者间一致性测量,量化了在随机机会之上获得的经验注释者一致性差异。我们提供了一个彻底的理论检查弗莱斯的卡帕之前,转向我们的推导DiPietro-Hazari卡帕。最后,我们总结了一个矩阵公式和一套程序指令,便于计算实现。

英文摘要:

Data is a key component of modern machine learning, but statistics for assessing data label quality remain sparse in literature. Here, we introduce DiPietro-Hazari Kappa, a novel statistical metric for assessing the quality of suggested dataset labels in the context of human annotation. Rooted in the classical Fleiss's Kappa measure of inter-annotator agreement, the DiPietro-Hazari Kappa quantifies the the empirical annotator agreement differential that was attained above random chance. We offer a thorough theoretical examination of Fleiss's Kappa before turning to our derivation of DiPietro-Hazari Kappa. Finally, we conclude with a matrix formulation and set of procedural instructions for easy computational implementation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑