多语言学习中存在数据集不平衡时顺序至关重要
Order Matters in the Presence of Dataset Imbalance for Multilingual Learning
AI总结：
针对多任务学习中数据不平衡问题，提出先高资源任务预训练、再高低资源混合任务微调的方法，经实证验证在NMT和多语言语言建模上优于标准静态加权。
摘要：
本文通过实证研究多任务学习的优化动态，重点关注管控存在显著数据不平衡的任务集合的优化机制。我们提出一种简单有效的方法：先在高资源任务上进行预训练，再在高/低资源任务的混合集上微调。我们对该方法的优势开展了全面实证研究与分析，结果表明，相较于标准静态加权的性能权衡曲线，该方法能实现持续提升。我们分析了该方法适用的数据场景，并在神经机器翻译（NMT）和多语言语言建模任务中通过实证验证了其改进效果。
英文摘要：
In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance. We present a simple yet effective method of pre-training on high-resource tasks, followed by fine-tuning on a mixture of high/low-resource tasks. We provide a thorough empirical study and analysis of this method's benefits showing that it achieves consistent improvements relative to the performance trade-off profile of standard static weighting. We analyze under what data regimes this method is applicable and show its improvements empirically in neural machine translation (NMT) and multi-lingual language modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
