用轻量级语言特定模块压缩多语言知识
Condensing Multilingual Knowledge with Lightweight Language-Specific Modules
AI总结：
本文提出语言特定矩阵合成（LMS）方法，通过生成低秩矩阵构建轻量级语言特定模块，并利用融合蒸馏技术压缩多语言知识，在减少参数的同时显著提升了多语言机器翻译性能。
摘要：
引入语言特定（LS）模块是提升多语言机器翻译性能的有效方法。由于该方法不会增加 FLOPs，因此与混合专家模型相似。然而，由于全连接层中的全秩矩阵引入了过多的参数，这种方法在扩展到数百种语言（专家）时往往难以管理。在本研究中，我们提出了语言特定矩阵合成（LMS）方法。该方法通过从两个显著更小的矩阵生成低秩矩阵来近似全秩矩阵，从而构建 LS 模块。此外，我们利用融合蒸馏（FD）技术，将来自多个 LS 模块的多语言知识压缩到一个共享模块中，以提高推理和模型序列化的效率。实验表明，在额外参数量相同的情况下，我们的 LMS 方法显著优于以往的 LS 方法和 MoE 方法，例如在多对多多语言机器翻译任务上，比 Switch Transformer 高出 1.73 个 BLEU 点。重要的是，LMS 能够以更少的参数实现相当的翻译性能。
英文摘要：
Incorporating language-specific (LS) modules is a proven method to boost performance in multilingual machine translation. This approach bears similarity to Mixture-of-Experts (MoE) because it does not inflate FLOPs. However, the scalability of this approach to hundreds of languages (experts) tends to be unmanageable due to the prohibitive number of parameters introduced by full-rank matrices in fully-connected layers. In this work, we introduce the Language-Specific Matrix Synthesis (LMS) method. This approach constructs LS modules by generating low-rank matrices from two significantly smaller matrices to approximate the full-rank matrix. Furthermore, we condense multilingual knowledge from multiple LS modules into a single shared module with the Fuse Distillation (FD) technique to improve the efficiency of inference and model serialization. We show that our LMS method significantly outperforms previous LS methods and MoE methods with the same amount of extra parameters, e.g., 1.73 BLEU points over the Switch Transformer on many-to-many multilingual machine translation. Importantly, LMS is able to have comparable translation performance with much fewer parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
