通过语言特定模型合并提升训练效率并降低维护成本
Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging
AI总结：
本文提出通过语言特定模型合并提升训练效率并降低维护成本，展示在多语言任务中显著减少训练时间和成本的方法。
摘要：
对任务特定的多语言大语言模型（LLM）进行微调，需要在包含所有所需语言示例的多语言数据集上训练模型。更新一个或多个支持语言以添加额外数据或添加新语言支持需要重新训练模型，这可能在计算上效率低下，并造成严重的维护瓶颈。最近关于合并多语言多任务模型的研究在质量提升方面显示出前景，但其计算和维护效率仍缺乏研究。在本工作中，我们从效率角度首次对这种合并策略进行了聚焦分析，评估了三个独立任务。我们展示了在保持质量相当的情况下，该合并方法能显著提升效率：这种合并方法可将初始训练时间减少高达50%。我们还展示了在模型维护中更新单个语言并重新合并可以降低训练成本，比重新训练完整的多语言模型降低超过60%。我们在公共和专有行业数据集上展示了这一点，证明了该方法在工业用例中的有效性，不仅适用于之前研究中已探讨的学术场景。
英文摘要：
Fine-tuning a task-specific multilingual large language model (LLM) involves training the model on a multilingual dataset with examples in all the required languages. Updating one or more supported languages with additional data or adding support for a new language involves retraining the model, which can be computationally inefficient and creates a severe maintenance bottleneck. Recent research on merging multilingual multitask models has shown promise in terms of improved quality, but its computational and maintenance efficiency remains unstudied. In this work, we provide the first focused analysis of this merging strategy from an efficiency perspective, evaluating it across three independent tasks. We demonstrate significant efficiency gains while maintaining parity in terms of quality: this merging approach reduces the initial training time by up to 50\%. We also demonstrate that updating an individual language and re-merging as part of model maintenance reduces training costs by more than 60\%, compared to re-training the full multilingual model. We show this on both public and proprietary industry datasets confirming that the approach works well for industrial use cases in addition to academic settings already studied in previous work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
