arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.14141cs.CL
收藏

Crowd-Calibrator:标注者分歧能否为主观任务的校准提供信息?

Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?

Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);Johns Hopkins University(约翰斯·霍普金斯大学);University of Southern California(南加利福尼亚大学)

Urja Khurana, Eric Nalisnick, Antske Fokkens, Swabha Swayamdipta

AI总结:

针对NLP主观任务中多数投票掩盖标注者分歧的问题,提出Crowd-Calibrator方法,通过建模众包工作者与模型标签分布的距离来校准模型并决定是否弃权(不执行),在仇恨言论检测和自然语言推理任务上表现优异。

摘要:

NLP中的主观任务大多被降级为客观标准,其中黄金标签由多数投票决定。这掩盖了标注者分歧和标签的固有不确定性。我们认为主观性应纳入模型决策,并在选择性预测设置下通过校准发挥直接作用。具体而言,我们不再纯粹从模型视角校准置信度,而是基于众包工作者的一致性对主观任务进行模型校准。我们的方法Crowd-Calibrator对众包工作者标签分布与模型自身标签分布之间的距离进行建模,以告知模型是否应弃权(不执行)决策。在仇恨言论检测和自然语言推理这两个高度主观的任务上,实验表明Crowd-Calibrator优于或与现有选择性预测基线取得竞争性表现。我们的发现强调了将人类决策引入模型预测的价值。

英文摘要:

Subjective tasks in NLP have been mostly relegated to objective standards, where the gold label is decided by taking the majority vote. This obfuscates annotator disagreement and the inherent uncertainty of the label. We argue that subjectivity should factor into model decisions and play a direct role via calibration under a selective prediction setting. Specifically, instead of calibrating confidence purely from the model's perspective, we calibrate models for subjective tasks based on crowd worker agreement. Our method, Crowd-Calibrator, models the distance between the distribution of crowd worker labels and the model's own distribution over labels to inform whether the model should abstain from a decision. On two highly subjective tasks, hate speech detection and natural language inference, our experiments show Crowd-Calibrator either outperforms or achieves competitive performance with existing selective prediction baselines. Our findings highlight the value of bringing human decision-making into model predictions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑