arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.07465cs.CL
收藏

关于对比评价中最小对的极限

On the Limits of Minimal Pairs in Contrastive Evaluation

University of Zurich(苏黎世大学);University of Edinburgh(爱丁堡大学)

Jannis Vamvas, Rico Sennrich

摘要:

最小句子对经常被用来分析语言模型的行为。通常认为,对比对上的模型行为可以预测模型的整体行为。我们认为,两个条件是必要的,这个假设成立:第一,一个测试的假设应该是动机良好,因为实验表明,对比评估可能会导致假阳性。其次,测试数据的选择应尽量减少评估时间和部署时间之间的分布差异。为了更好地近似部署时解码,我们建议基于机器生成的文本创建最小对,而不是人类编写的引用。我们提出了一个对比评估套件,实现这一建议的英语-德语MT。

英文摘要:

Minimal sentence pairs are frequently used to analyze the behavior of language models. It is often assumed that model behavior on contrastive pairs is predictive of model behavior at large. We argue that two conditions are necessary for this assumption to hold: First, a tested hypothesis should be well-motivated, since experiments show that contrastive evaluation can lead to false positives. Secondly, test data should be chosen such as to minimize distributional discrepancy between evaluation time and deployment time. For a good approximation of deployment-time decoding, we recommend that minimal pairs are created based on machine-generated text, as opposed to human-written references. We present a contrastive evaluation suite for English-German MT that implements this recommendation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑