arXiv 2407.058412024-10-22cs.CLcs.LG

语言模型词汇扩展与初始化方法的实证比较

An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

Indian Institute of Technology Madras（印度马德拉斯理工学院）；National Institute of Information and Communications Technology, Japan（日本信息通信研究机构）；Indian Institute of Technology Bombay（印度孟买理工学院）；IT University of Copenhagen, Denmark（丹麦哥本哈根信息技术大学）；Microsoft India（微软印度公司）；Nilekani Centre at AI4Bharat（AI4Bharat的尼莱卡尼中心）

Nandini Mundra, Aditya Nanda Kishore, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra

备注：: CONLL 2024 (EMNLP 2024)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文针对多语言语言模型分词器词汇扩展中的嵌入初始化问题，提出无需跨语言嵌入的约束 Word2Vec（CW2V）方法，并在 RoBERTa 和 LLaMA 2 上验证其与先进方法相当甚至更优。

摘要：

语言模型（LMs）在英语自然语言处理任务中表现出色，但在大多数其他语言中性能有所下降。这一问题通常通过针对这些语言进行持续预训练和微调来解决。该过程中的一个重要问题是原始模型分词器的词汇覆盖有限，导致对新语言的表示不足，因而需要扩展分词器。与新词汇项对应嵌入的初始化带来了进一步挑战。当前策略需要跨语言嵌入，且缺乏坚实的理论基础以及与强基线的比较。在本文中，我们首先从理论上证明，在现有嵌入的凸包内进行初始化是一种良好的初始化方式，随后提出一种新颖而简单的方法——约束 Word2Vec（CW2V），该方法不需要跨语言嵌入。我们的研究评估了在四种语言和五个任务上扩展 RoBERTa 和 LLaMA 2 的不同初始化方法。结果表明，CW2V 与更先进的技术表现相当甚至更好。此外，多元初始化等更简单的方法与这些先进方法表现相当，表明即使采用更简单的初始化方法，也能实现高效的大规模多语言持续预训练。我们公开发布了代码（https://github.com/AI4Bharat/VocabAdaptation_LLM/tree/CW2V）。

英文摘要：

Language Models (LMs) excel in natural language processing tasks for English but show reduced performance in most other languages. This problem is commonly tackled by continually pre-training and fine-tuning these models for said languages. A significant issue in this process is the limited vocabulary coverage in the original model's tokenizer, leading to inadequate representation of new languages and necessitating an expansion of the tokenizer. The initialization of the embeddings corresponding to new vocabulary items presents a further challenge. Current strategies require cross-lingual embeddings and lack a solid theoretical foundation as well as comparisons with strong baselines. In this paper, we first establish theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V), which does not require cross-lingual embeddings. Our study evaluates different initialization methods for expanding RoBERTa and LLaMA 2 across four languages and five tasks. The results show that CW2V performs equally well or even better than more advanced techniques. Additionally, simpler approaches like multivariate initialization perform on par with these advanced methods indicating that efficient large-scale multilingual continued pretraining can be achieved even with simpler initialization methods. We release our code publicly (https://github.com/AI4Bharat/VocabAdaptation_LLM/tree/CW2V).

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。