arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.21140cs.CLcs.AI
收藏

利用学习嵌入传播促进大语言模型的俄语适配

Facilitating large language model Russian adaptation with Learned Embedding Propagation

Lomonosov Moscow State University(莫斯科罗蒙诺索夫国立大学)

Mikhail Tikhomirov, Daniil Chernyshev

AI总结:

本文提出学习嵌入传播(LEP)方法,通过直接向现有指令微调LLM植入新语言知识,以低数据需求实现俄语适配,性能媲美传统指令微调方法。

摘要:

大语言模型(LLM)技术的快速进步催生了强大的开源指令微调LLM,这些模型具有与GPT-4等最先进同类模型相当的文本生成质量。尽管此类模型的出现加速了LLM技术在敏感信息环境中的采用,但这些模型的作者并未公开复现结果所需的训练数据,从而使这些成果成为模型专属。由于这些开源模型也是多语言的,这反过来又降低了训练特定语言LLM的收益,因为推理计算效率的提升成为这一昂贵过程中唯一有保障的优势。更经济高效的选择,如词汇表扩展和后续持续预训练,也因缺乏高质量指令微调数据而受到抑制,因为指令微调数据是决定LLM任务解决能力的主要因素。为了解决这些局限性并降低语言适配流程的成本,我们提出了学习嵌入传播(LEP)。与现有方法不同,我们的方法对现有LLM知识的影响极小,因此训练数据规模需求更低,我们通过新颖的专用嵌入传播程序来强化这一特性,该程序允许跳过指令微调步骤,直接将新的语言知识植入任何现有的指令微调变体中。我们评估了LLaMa-3-8B和Mistral-7B的四种俄语词汇适配方案,结果表明LEP与传统指令微调方法相比具有竞争力,其性能可与OpenChat 3.5和LLaMa-3-8B-Instruct相媲美,并通过自校准和持续微调进一步提升了任务解决能力。

英文摘要:

Rapid advancements of large language model (LLM) technologies led to the introduction of powerful open-source instruction-tuned LLMs that have the same text generation quality as the state-of-the-art counterparts such as GPT-4. While the emergence of such models accelerates the adoption of LLM technologies in sensitive-information environments the authors of such models don not disclose the training data necessary for replication of the results thus making the achievements model-exclusive. Since those open-source models are also multilingual this in turn reduces the benefits of training a language specific LLMs as improved inference computation efficiency becomes the only guaranteed advantage of such costly procedure. More cost-efficient options such as vocabulary extension and subsequent continued pre-training are also inhibited by the lack of access to high-quality instruction-tuning data since it is the major factor behind the resulting LLM task-solving capabilities. To address the limitations and cut the costs of the language adaptation pipeline we propose Learned Embedding Propagation (LEP). Unlike existing approaches our method has lower training data size requirements due to minimal impact on existing LLM knowledge which we reinforce using novel ad-hoc embedding propagation procedure that allows to skip the instruction-tuning step and instead implant the new language knowledge directly into any existing instruct-tuned variant. We evaluated four Russian vocabulary adaptations for LLaMa-3-8B and Mistral-7B, showing that LEP is competitive with traditional instruction-tuning methods, achieving performance comparable to OpenChat 3.5 and LLaMa-3-8B-Instruct, with further improvements via self-calibration and continued tuning enhancing task-solving capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑