arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.09422cs.CL
收藏

利用正确性判别器预测泛化性能

Predicting generalization performance with correctness discriminators

Department of Language Science and Technology(语言科学与技术系);Saarland Informatics Campus(萨尔兰州信息校园);Saarland University(萨尔兰大学)

Yuekun Yao, Alexander Koller

AI总结:

针对NLP模型泛化性能预测难题,提出基于正确性判别器的方法,无需未见过数据的金标签即可给出准确率上下界,在多类任务中验证了界的可靠性与紧密性。

摘要:

预测NLP模型在未见过的、可能分布外的数据上的准确率,是实现可信度的前提。我们提出一种新型模型,无需未见过数据的金标签,就能确定准确率的上下界。我们通过训练一个判别器来实现这一点,该判别器预测给定序列到序列模型的输出是否正确。我们在多种标注、句法分析和语义解析任务上验证,金标准准确率可靠地处于预测的上下界之间,且这些上下界的差距非常小。

英文摘要:

The ability to predict an NLP model's accuracy on unseen, potentially out-of-distribution data is a prerequisite for trustworthiness. We present a novel model that establishes upper and lower bounds on the accuracy, without requiring gold labels for the unseen data. We achieve this by training a discriminator which predicts whether the output of a given sequence-to-sequence model is correct or not. We show across a variety of tagging, parsing, and semantic parsing tasks that the gold accuracy is reliably between the predicted upper and lower bounds, and that these bounds are remarkably close together.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑