arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.09644cs.CL
收藏

用于改变 LLM 词汇表的适配器:哪些语言受益最大?

Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?

University of Maryland(马里兰大学);Microsoft(微软公司);Amazon(亚马逊公司)

HyoJung Han, Akiko Eriguchi, Haoran Xu, Hieu Hoang, Marine Carpuat, Huda Khayrallah

AI总结:

本文提出 VocADT,用固定模型权重下的适配器学习现有嵌入的最优线性组合以扩展 LLM 词汇表;11 种语言实验表明其在多语言理解和翻译任务上优于 Mistral 等基线。

摘要:

词汇表适配将新词汇整合到预训练语言模型中,能够支持向新语言扩展,并缓解 token 过度碎片化问题。然而,现有方法受限于对启发式规则或外部嵌入的依赖。我们提出 VocADT,这是一种使用适配器模块进行词汇表适配的新方法;这些适配器经过训练,可在保持模型权重固定的同时,学习现有嵌入的最优线性组合。VocADT 提供了灵活且可扩展的解决方案,不依赖外部资源或语言约束。在涵盖多样文字、资源可用性和碎片化程度的 11 种语言上,我们证明 VocADT 在包括自然语言理解和机器翻译在内的多种多语言任务中优于原始 Mistral 模型及其他基线。我们发现,拉丁文字语言和高度碎片化语言从词汇表适配中受益最大。我们进一步在机器翻译这一生成任务上微调适配后的模型,发现词汇表适配在微调后仍然有益,并且 VocADT 最为有效。

英文摘要:

Vocabulary adaptation, which integrates new vocabulary into pre-trained language models, enables expansion to new languages and mitigates token over-fragmentation. However, existing approaches are limited by their reliance on heuristics or external embeddings. We propose VocADT, a novel method for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model's weights fixed. VocADT offers a flexible and scalable solution without depending on external resources or language constraints. Across 11 languages-with diverse scripts, resource availability, and fragmentation-we demonstrate that VocADT outperforms the original Mistral model and other baselines across various multilingual tasks including natural language understanding and machine translation. We find that Latin-script languages and highly fragmented languages benefit the most from vocabulary adaptation. We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adaptation is still beneficial after fine-tuning and that VocADT is the most effective.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑