arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.08676cs.CLcs.AIcs.LG
收藏

Distributed NLI:学习预测语言推理的人类意见分布

Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning

University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)

Xiang Zhou, Yixin Nie, Mohit Bansal

摘要:

我们介绍了分布式NLI,一个新的NLU任务的目标是预测分布的人类判断自然语言推理。我们表明,通过应用额外的分布估计方法,即蒙特卡罗(MC)辍学,深度增强,重新校准和分布蒸馏,模型可以比softmax基线更有效地捕获人类判断分布。我们表明,MC辍学是能够实现体面的性能,没有任何分布注释,而重新校准可以提供额外的分布注释进一步改进,建议多个注释的价值为一个例子,在模拟人类判断的分布。尽管有这些改进,最好的结果仍然远远低于估计的人类上限,这表明预测人类判断的分布仍然是一个开放的,具有挑战性的问题,有很大的改进空间。我们展示了MC丢弃和重新校准的常见错误。最后,我们给出了使用这些方法的指导方针与不同级别的数据可用性,并鼓励未来的工作建模的人类意见分布的语言推理。我们的代码和数据可在https://github.com/easonnie/ChaosNLI上公开获取

英文摘要:

We introduce distributed NLI, a new NLU task with a goal to predict the distribution of human judgements for natural language inference. We show that by applying additional distribution estimation methods, namely, Monte Carlo (MC) Dropout, Deep Ensemble, Re-Calibration, and Distribution Distillation, models can capture human judgement distribution more effectively than the softmax baseline. We show that MC Dropout is able to achieve decent performance without any distribution annotations while Re-Calibration can give further improvements with extra distribution annotations, suggesting the value of multiple annotations for one example in modeling the distribution of human judgements. Despite these improvements, the best results are still far below the estimated human upper-bound, indicating that predicting the distribution of human judgements is still an open, challenging problem with a large room for improvements. We showcase the common errors for MC Dropout and Re-Calibration. Finally, we give guidelines on the usage of these methods with different levels of data availability and encourage future work on modeling the human opinion distribution for language reasoning. Our code and data are publicly available at https://github.com/easonnie/ChaosNLI

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑