多语言神经机器翻译的帕累托前沿研究
On the Pareto Front of Multilingual Neural Machine Translation
AI总结：
本文通过训练200多个多语言模型发现MNMT中数据不平衡导致标量化偏离传统帕累托前沿，提出双幂律预测性能权衡前沿，并据此优化采样比例，以更少训练预算超越温度搜索和梯度操纵方法。
摘要：
在这项工作中，我们研究了在多语言神经机器翻译（MNMT）中，给定方向的性能如何随其采样比例变化。通过训练超过200个具有不同模型规模、数据规模和语言方向的多语言模型，我们发现一个有趣的现象：某些翻译方向的性能并不总是随着其在多任务优化目标中权重的增加而提升。相应地，当训练语料中存在数据不平衡时，标量化方法会导致多任务权衡前沿偏离传统的帕累托前沿，这对提升所有方向的整体性能构成了巨大挑战。基于我们的观察，我们提出了双幂律（Double Power Law）来预测MNMT中独特的性能权衡前沿，该方法在各种语言、数据充足程度和任务数量下均具有鲁棒性。最后，我们将MNMT中的采样比例选择问题形式化为一个基于双幂律的优化问题。在我们的实验中，该方法仅使用总训练预算的1/5到1/2，就取得了优于温度搜索和梯度操纵方法的性能。我们已在 https://github.com/pkunlp-icler/ParetoMNMT 发布代码以供复现。
英文摘要：
In this work, we study how the performance of a given direction changes with its sampling ratio in Multilingual Neural Machine Translation (MNMT). By training over 200 multilingual models with various model sizes, data sizes, and language directions, we find it interesting that the performance of certain translation direction does not always improve with the increase of its weight in the multi-task optimization objective. Accordingly, scalarization method leads to a multitask trade-off front that deviates from the traditional Pareto front when there exists data imbalance in the training corpus, which poses a great challenge to improve the overall performance of all directions. Based on our observations, we propose the Double Power Law to predict the unique performance trade-off front in MNMT, which is robust across various languages, data adequacy, and the number of tasks. Finally, we formulate the sample ratio selection problem in MNMT as an optimization problem based on the Double Power Law. In our experiments, it achieves better performance than temperature searching and gradient manipulation methods with only 1/5 to 1/2 of the total training budget. We release the code at https://github.com/pkunlp-icler/ParetoMNMT for reproduction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
