arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.06859cs.CL
收藏

捕获标签分布:NLI中的案例研究

Capturing Label Distribution: A Case Study in NLI

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Shujian Zhang, Chengyue Gong, Eunsol Choi

摘要:

我们研究估计固有的人类分歧(注释标签分布)在自然语言推理任务。预测标签分布的事后平滑以匹配预期标签熵是非常有效的。这种简单的操作可以将KL分歧减少近一半,但不会提高大多数标签预测的准确性或学习标签分布。为此,我们在训练中引入了少量具有多个引用的示例。我们从每个训练样本收集单个引用的标准实践出发,发现在固定的注释预算下,收集多个引用可以实现更好的准确性。最后,我们提供了丰富的分析比较这两种方法,以改善标签分布估计。

英文摘要:

We study estimating inherent human disagreement (annotation label distribution) in natural language inference task. Post-hoc smoothing of the predicted label distribution to match the expected label entropy is very effective. Such simple manipulation can reduce KL divergence by almost half, yet will not improve majority label prediction accuracy or learn label distributions. To this end, we introduce a small amount of examples with multiple references into training. We depart from the standard practice of collecting a single reference per each training example, and find that collecting multiple references can achieve better accuracy under the fixed annotation budget. Lastly, we provide rich analyses comparing these two methods for improving label distribution estimation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑