通过激进剪枝专家从LLM中提取小型翻译专家
Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
AI总结：
提出一种从混合专家LLM中激进剪枝与翻译无关的专家，实现大幅压缩MoE块而不显著降低翻译质量的方法。
摘要：
现代大型语言模型（LLM）实现了最先进的机器翻译性能，但它们是作为广泛通才训练的，主要针对许多与翻译无关的任务和能力。因此，它们对于此任务严重过参数化，导致过多的内存和计算需求。在本文中，我们提出了一种从现代混合专家LLM中激进剪枝专家的方法，同时翻译质量下降可忽略不计。我们的方法利用专家专业化和LLM中多语言能力的可分离性来识别与翻译无关的专家。并且由于MoE的模块化特性，这些专家可以在无需任何训练的情况下轻松剪枝。无需重新训练，我们能够剪枝一半的专家而质量下降可忽略，剪枝70%仅造成轻微损失。通过非常短的SFT，我们剪枝75%的专家并恢复基线性能，在某些设置下移除近90%的专家同时保持合理的翻译质量。总体而言，我们的结果表明翻译仅需要LLM的一小部分，从而实现了对包含超过90%参数的MoE块的大幅压缩。
英文摘要：
Modern large language models (LLMs) achieve state-of-the-art machine translation performance, but they do so as broad generalists largely trained for many tasks and capabilities unrelated to translation. Thus, they are heavily overparameterized for this task, resulting in excessive memory and compute requirements. In this paper, we present a method for aggressively pruning experts from modern mixture-of-experts LLMs while incurring negligible degradation in translation quality. Our approach exploits expert specialization and the separability of multilingual capabilities in LLMs to identify experts irrelevant to translation. And because of the modular nature of MoEs, these can be easily pruned without any training. Without retraining, we are able to prune half of all experts with negligible degradation and 70% with only minor losses. With a very short SFT, we prune 75% of experts while recovering baseline performance, and in some settings remove nearly 90% while maintaining reasonable translation quality. Overall, our results show that translation requires only a fraction of the LLM, enabling substantial compression of the MoE blocks that contain over 90% of parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
