arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.02570cs.CLcs.LG
收藏

人类标签变异的“问题”:数据、建模和评估中的真值

The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation

Center for Information and Language Processing (CIS)(信息与语言处理中心(CIS));LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心(MCML))

Barbara Plank

摘要:

标签中的人为变化通常被认为是噪音。机器学习(ML)的注释项目旨在最大限度地减少人类标签的变化,并假设最大限度地提高数据质量,进而优化和最大限度地提高机器学习指标。然而,这种常规实践假设存在真值,并且忽略了由于不一致、注释中的主观性或多个合理答案而在标记中存在真正的人类变化。在这份立场文件中,我们认为,人类标签变异的这个大问题仍然存在,迫切需要更多的关注,以推动我们的领域向前发展。这是因为人类标签的变化会影响ML管道的所有阶段:数据,建模和评估。然而,很少有作品认为所有这些方面联合,现有的研究是支离破碎的。我们调和了以前提出的不同的人类标签变异概念,提供了一个公开可用的数据集与未聚合的标签,描述了迄今为止提出的方法,确定差距,并提出了前进的道路。随着数据集变得越来越可用,我们希望这种对“问题”的综合看法将导致对可能的战略进行公开讨论,以制定根本性的新方向。

英文摘要:

Human variation in labeling is often considered noise. Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics. However, this conventional practice assumes that there exists a ground truth, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers. In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward. This is because human label variation impacts all stages of the ML pipeline: data, modeling and evaluation. However, few works consider all of these dimensions jointly; and existing research is fragmented. We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward. As datasets are becoming increasingly available, we hope that this synthesized view on the 'problem' will lead to an open discussion on possible strategies to devise fundamentally new directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑