arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.03477cs.CLcs.AI
收藏

Tik-to-Tok: 逐词元翻译语言模型：一种用于高效语言适配的嵌入初始化策略

Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

Ghent University（根特大学）；imec（imec 微电子研究中心）；KU Leuven（鲁汶大学）

François Remy, Pieter Delobelle, Bettina Berendt, Kris Demuynck, Thomas Demeester

备注：
As first reviewed at TACL
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对中低资源语言预训练数据不足的问题，提出基于一对多词元映射的嵌入初始化模型转换策略，将高资源单语模型适配到目标语言，在荷兰语、弗里斯兰语下游任务上取得SOTA，降低训练的数据与时间成本。

摘要：

为低资源和中等资源语言训练单语语言模型面临着预训练数据有限且往往不足的挑战。在本研究中，我们提出了一种新颖的模型转换策略来解决这一问题，将高资源单语语言模型适配到新的目标语言。通过对涵盖源语言和目标语言的单词翻译词典进行泛化，我们将目标语言分词器的词元映射到源语言分词器中语义相似的词元。这种一对多的词元映射极大地改善了目标语言嵌入表的初始化效果。我们开展了将高资源模型转换为中等资源和低资源语言（即荷兰语和弗里斯兰语）的实验。这些转换后的模型在这些语言的各类下游任务上均取得了新的state-of-the-art（最优）性能。通过大幅减少训练最优模型所需的数据量和时间，我们提出的新型模型转换策略有望惠及全球众多语言。

英文摘要：

Training monolingual language models for low and mid-resource languages is made challenging by limited and often inadequate pretraining data. In this study, we propose a novel model conversion strategy to address this issue, adapting high-resources monolingual language models to a new target language. By generalizing over a word translation dictionary encompassing both the source and target languages, we map tokens from the target tokenizer to semantically similar tokens from the source language tokenizer. This one-to-many token mapping improves tremendously the initialization of the embedding table for the target language. We conduct experiments to convert high-resource models to mid- and low-resource languages, namely Dutch and Frisian. These converted models achieve a new state-of-the-art performance on these languages across all sorts of downstream tasks. By reducing significantly the amount of data and time required for training state-of-the-art models, our novel model conversion strategy has the potential to benefit many languages worldwide.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑