Human variation in labeling is often considered noise. Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics. However, this conventional practice assumes that there exists a ground truth, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers. In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward. This is because human label variation impacts all stages of the ML pipeline: data, modeling and evaluation. However, few works consider all of these dimensions jointly; and existing research is fragmented. We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward. As datasets are becoming increasingly available, we hope that this synthesized view on the 'problem' will lead to an open discussion on possible strategies to devise fundamentally new directions.
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Seoul National University(首尔大学);NAVER AI Lab(NAVER人工智能实验室);NAVER CLOVA;Korea Advanced Institute of Science and Technology(韩国科学技术院);Hanyang University(汉阳大学)
An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
Brigham Young University(杨百翰大学)
预先训练的语言模型从它们所训练的大量语料库中获得大量的语言和事实知识,并且提示工程试图将这些模型与特定任务相匹配。不幸的是,现有的提示工程方法需要大量的标记数据,访问模型参数,或两者兼而有之。本文介绍了一种新的选择提示模板\textit{without labeled examples}和\textit{without direct access to the model}的方法。具体来说,在一组候选模板,我们选择的模板,最大化的互信息之间的输入和相应的模型输出。在8个数据集,代表7个不同的NLP任务,我们表明,当一个模板具有高互信息,它也有高精度的任务。在最大的模型上,使用我们的方法选择提示可以获得从平均提示准确度到最佳提示准确度的90%,并且不需要真值标签。
Mitigating Shortcuts in Language Models with Soft Label Encoding
Stevens Institute of Technology(史蒂文斯理工学院);Shanghai Jiao Tong University(上海交通大学);New Jersey Institute of Technology(新泽西理工学院);University of Georgia(佐治亚大学)
Wisdom of the AI Crowd (AI-CROWD) for Ground Truth Approximation in Content Analysis: A Research Protocol & Validation Using Eleven Large Language Models
University of Alcalá(阿尔卡拉大学);Carlos III University of Madrid(马德里卡洛斯三世大学)
Beyond Single Ground Truth: Reference Monism as Epistemic Injustice in ASR Evaluation
Department of Information Science, Cornell University(康奈尔大学信息科学系);Department of Computer Science, Texas A&M University(德克萨斯农工大学计算机科学系);Rev AI;Tundra Technical Solutions(Tundra技术解决方案);Division of Liberal Studies, Kangwon National University(康sworth国立大学文科系)
Ground Truth Generation for Multilingual Historical NLP using LLMs
ARTFL Project, Romance Languages and Literatures, University of Chicago(ARTFL项目, Romance语言与文学系,芝加哥大学);Department of History, University of Chicago(历史系,芝加哥大学);Libraries and School of Information Studies, Purdue University(图书馆与信息科学学院,普渡大学)