arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.15217cs.CLcs.AI
收藏

挖掘NMT中的错误:从部分假设空间评估和理解模型错误

Digging Errors in NMT: Evaluating and Understanding Model Errors from Partial Hypothesis Space

Tencent(腾讯);WeChat AI, Tencent(腾讯微信人工智能实验室)

Jianhao Yan, Chenming Wu, Fandong Meng, Jie Zhou

摘要:

对神经机器翻译(NMT)进行可靠的评价是理解和改进NMT的关键。NMT系统的当前评估通常建立在启发式解码算法(例如,束搜索)和评估翻译和黄金参考之间的相似性的评估度量。然而,这种系统级评估框架仅评估一个最佳假设和启发式解码算法带来的搜索错误。为了更好地理解NMT模型,我们提出了一种新的评估协议,该协议定义了模型误差与模型在假设空间上的排名能力。为了解决指数大空间的问题,我们提出了两种近似方法,顶部区域评估以及一个精确的顶部- $k$解码算法,它发现在整个假设空间中排名靠前的假设,和蒙特卡洛抽样评估,从更广泛的角度模拟假设空间。为了量化误差,我们通过测量模型排名的假设阵列与理想排名的假设阵列之间的距离来定义NMT模型误差。在确认与人类判断的强相关性后,我们将我们的评估应用于各种NMT基准和模型架构。我们发现,国家的最先进的Transformer模型面临严重的排名问题,只有在顶部区域的随机机会水平。我们进一步分析了具有不同深度和宽度的架构上的模型误差,以及不同的数据增强技术,展示了这些因素如何影响模型误差。最后,我们将模型误差与搜索算法联系起来,并提供了束搜索归纳偏差和最小贝叶斯风险(MBR)解码相关性的有趣发现。

英文摘要:

Solid evaluation of neural machine translation (NMT) is key to its understanding and improvement. Current evaluation of an NMT system is usually built upon a heuristic decoding algorithm (e.g., beam search) and an evaluation metric assessing similarity between the translation and golden reference. However, this system-level evaluation framework is limited by evaluating only one best hypothesis and search errors brought by heuristic decoding algorithms. To better understand NMT models, we propose a novel evaluation protocol, which defines model errors with model's ranking capability over hypothesis space. To tackle the problem of exponentially large space, we propose two approximation methods, top region evaluation along with an exact top-$k$ decoding algorithm, which finds top-ranked hypotheses in the whole hypothesis space, and Monte Carlo sampling evaluation, which simulates hypothesis space from a broader perspective. To quantify errors, we define our NMT model errors by measuring distance between the hypothesis array ranked by the model and the ideally ranked hypothesis array. After confirming the strong correlation with human judgment, we apply our evaluation to various NMT benchmarks and model architectures. We show that the state-of-the-art Transformer models face serious ranking issues and only perform at the random chance level in the top region. We further analyze model errors on architectures with different depths and widths, as well as different data-augmentation techniques, showing how these factors affect model errors. Finally, we connect model errors with the search algorithms and provide interesting findings of beam search inductive bias and correlation with Minimum Bayes Risk (MBR) decoding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑