Franken-Adapter：通过嵌入手术实现大语言模型的跨语言适配
Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery
AI总结：
针对LLMs低资源语言能力不足问题，提出Franken-Adapter模块化适配方法，通过嵌入手术定制目标语言词汇并调优嵌入，结合英语指令调优模型实现零样本跨语言迁移，在多语言任务中显著提升性能且英语退化小，还能迁移推理能力。
摘要：
大型语言模型（LLMs）在低资源语言中的能力远落后于英语，使其普及性面临重大挑战。为此，我们提出Franken-Adapter，这是一种针对仅解码器LLMs的模块化语言适配方法，包含嵌入手术。该方法首先为目标语言创建定制词汇表，通过多语言数据上的嵌入调优进行语言适配；随后将预训练嵌入与在英语对齐数据上指令调优的LLMs集成，实现零样本跨语言迁移。在参数达27B的Gemma2模型上的实验显示，96种语言在判别式和生成式任务上提升高达20%，英语性能退化极小（<1%）。深入分析表明定制分词器对增强语言适配和提升推理效率至关重要；此外，该方法在20种语言上较数学优化LLM提升14%，为事后跨语言迁移推理能力提供模块化解决方案。
英文摘要：
The capabilities of Large Language Models (LLMs) in low-resource languages lag far behind those in English, making their universal accessibility a significant challenge. To alleviate this, we present $\textit{Franken-Adapter}$, a modular language adaptation approach for decoder-only LLMs with embedding surgery. Our method begins by creating customized vocabularies for target languages and performing language adaptation through embedding tuning on multilingual data. These pre-trained embeddings are subsequently integrated with LLMs that have been instruction-tuned on English alignment data to enable zero-shot cross-lingual transfer. Our experiments on $\texttt{Gemma2}$ models with up to 27B parameters demonstrate improvements of up to 20% across 96 languages, spanning both discriminative and generative tasks, with minimal regressions ($<$1%) in English. Further in-depth analysis reveals the critical role of customizing tokenizers in enhancing language adaptation, while boosting inference efficiency. Additionally, we show the versatility of our method by achieving a 14% improvement over a math-optimized LLM across 20 languages, offering a modular solution to transfer reasoning abilities across languages post hoc.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
