arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2203.11413cs.CLcs.AI
收藏

基于transformer的神经机器翻译的学习置信度

Learning Confidence for Transformer-based Neural Machine Translation

School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tencent(腾讯)

Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, Mu Li

摘要:

置信度估计旨在量化模型预测的置信度,提供成功的预期。当在现实环境中给定噪声样本和分布外数据时,经过良好校准的置信度估计可以实现准确的故障预测和适当的风险测量。然而,这一任务对于神经机器翻译(NMT)仍然是一个严峻的挑战,其中来自softmax分布的概率无法描述模型何时可能出错。为了解决这个问题,我们提出了一个无监督的置信度估计学习与NMT模型的训练。我们将置信度解释为NMT模型需要多少提示才能做出正确的预测,提示越多表明置信度越低。具体来说,NMT模型可以选择要求提示来提高翻译准确性,但代价是一些轻微的惩罚。然后,我们通过计算模型使用的提示数量来近似他们的置信水平。我们证明了我们学习的置信度估计在广泛的句子/单词级质量估计任务中达到了很高的准确性。分析结果验证了我们的置信度估计可以正确评估两种真实场景中的潜在风险:(1)发现噪声样本和(2)检测域外数据。我们进一步提出了一种新的基于置信度的特定于实例的标签平滑方法,该方法基于我们学习的置信度估计,其性能优于标准标签平滑。

英文摘要:

Confidence estimation aims to quantify the confidence of the model prediction, providing an expectation of success. A well-calibrated confidence estimate enables accurate failure prediction and proper risk measurement when given noisy samples and out-of-distribution data in real-world settings. However, this task remains a severe challenge for neural machine translation (NMT), where probabilities from softmax distribution fail to describe when the model is probably mistaken. To address this problem, we propose an unsupervised confidence estimate learning jointly with the training of the NMT model. We explain confidence as how many hints the NMT model needs to make a correct prediction, and more hints indicate low confidence. Specifically, the NMT model is given the option to ask for hints to improve translation accuracy at the cost of some slight penalty. Then, we approximate their level of confidence by counting the number of hints the model uses. We demonstrate that our learned confidence estimate achieves high accuracy on extensive sentence/word-level quality estimation tasks. Analytical results verify that our confidence estimate can correctly assess underlying risk in two real-world scenarios: (1) discovering noisy samples and (2) detecting out-of-domain data. We further propose a novel confidence-based instance-specific label smoothing approach based on our learned confidence estimate, which outperforms standard label smoothing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑