arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.11318cs.AIcs.CLcs.CY
收藏

共识陷阱:数据标注中主观性与‘真实真相’幻觉的剖析

The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation

University of Toronto Computer Science(多伦多大学计算机科学系);University of Toronto Engineering Science(多伦多大学工程科学系);Carnegie Mellon University School of Computer Science(卡内基梅隆大学计算机科学学院);Yale University American Studies(耶鲁大学美国研究系);University of Waterloo Computer Science(滑铁卢大学计算机科学系);Google Research(谷歌研究);University of Toronto(多伦多大学);Carnegie Mellon University(卡内基梅隆大学);Yale University(耶鲁大学);University of Waterloo(滑铁卢大学)

Sheza Munir, Benjamin Mah, Krisha Kalsi, Shivani Kapania, Julian Posada, Edith Law, Ding Wang, Syed Ishtiaque Ahmed

AI总结:

本文通过分析2020-2025年间七大赛事的346篇论文,揭示数据标注实践中‘共识陷阱’的机制,指出位置可读性失效与模型中介标注导致的锚定偏见,以及地理霸权对西方标准的强制性,呼吁将分歧视为高保真信号以构建文化胜任模型。

摘要:

在机器学习中,‘真实真相’指用于训练和评估模型的假设正确标签。然而,基础的‘真实真相’范式基于一种乐观谬误,将人类分歧视为技术噪声而非关键的社技术信号。本文通过系统文献综述分析2020-2025年间在ACL、AIES、CHI、CSCW、EAAMO、FAcT和NeurIPS等七个顶级会议发表的346篇论文,研究数据标注实践中促成‘共识陷阱’的机制。我们的反思性主题分析揭示,位置可读性系统的失败,结合近期向人类作为验证者的模型转变,特别是对模型中介标注的依赖,引入了深层次的锚定偏见,并有效排除了人类声音的参与。我们进一步展示地理霸权如何将西方规范强加为普遍标准,通常通过不稳定的数据显示工作者的表演性对齐来实施,这些工作者优先考虑请求者合规性而非诚实主观性以避免经济惩罚。批判‘噪声传感器’谬误,即统计模型将多元主义误诊为错误,我们主张重新夺回分歧作为高保真信号,以构建文化胜任模型。为解决这些系统张力,我们提出一个路线图,为多元标注基础设施提供指导,将目标从发现单一‘正确’答案转向映射人类经验的多样性。

英文摘要:

In machine learning, "ground truth" refers to the assumed correct labels used to train and evaluate models. However, the foundational "ground truth" paradigm rests on a positivistic fallacy that treats human disagreement as technical noise rather than a vital sociotechnical signal. This systematic literature review analyzes research published between 2020 and 2025 across seven premier venues: ACL, AIES, CHI, CSCW, EAAMO, FAccT, and NeurIPS, investigating the mechanisms in data annotation practices that facilitate this "consensus trap". Our reflexive thematic analysis of 346 papers reveals that systemic failures in positional legibility, combined with the recent architectural shift toward human-as-verifier models, specifically the reliance on model-mediated annotations, introduce deep-seated anchoring bias and effectively remove human voices from the loop. We further demonstrate how geographic hegemony imposes Western norms as universal benchmarks, often enforced by the performative alignment of precarious data workers who prioritize requester compliance over honest subjectivity to avoid economic penalties. Critiquing the "noisy sensor" fallacy, where statistical models misdiagnose pluralism as error, we argue for reclaiming disagreement as a high-fidelity signal essential for building culturally competent models. To address these systemic tensions, we propose a roadmap for pluralistic annotation infrastructures that shift the objective from discovering a singular "right" answer to mapping the diversity of human experience.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑