零样本分词器迁移
Zero-Shot Tokenizer Transfer
AI总结：
本研究提出零样本分词器迁移问题ZeTT，通过训练预测词元嵌入的超网络实现分词器即插即用，在跨语言、编程任务接近原模型性能且缩短序列长度，还可适配微调模型，推动LM与分词器解耦。
摘要：
语言模型（LM）与其分词器绑定，分词器将原始文本映射为词汇项（词元）序列。这限制了模型的灵活性：例如，主要以英语训练的语言模型在其他自然语言和编程语言中或许仍能表现良好，但由于其以英语为中心的分词器，效率会大幅下降。为缓解这一问题，我们应当能够在运行时将原语言模型的分词器替换为任意分词器，且不降低性能。因此，本研究定义了一个新问题：零样本分词器迁移（Zero-Shot Tokenizer Transfer，ZeTT）。ZeTT的核心挑战在于为新分词器词表中的词元寻找嵌入表示。由于现有的嵌入初始化启发式方法在ZeTT场景下往往表现处于随机水平，我们提出了一种新的解决方案：训练一个以分词器为输入、预测对应嵌入的超网络。我们通过实验证明，该超网络可泛化到编码器型（如XLM-R）和解码器型大语言模型（如Mistral-7B）的新分词器上。我们的方法在跨语言和编程任务中接近原模型的性能，同时显著缩短了分词后的序列长度。我们还发现，仅需用不足10亿个词元进行持续训练，就能快速弥补剩余的性能差距。最后，我们表明为基础（大）语言模型训练的ZeTT超网络，无需额外训练即可应用于微调后的变体。总体而言，我们的研究结果在将语言模型与其分词器解耦的方向上迈出了实质性步伐。
英文摘要：
Language models (LMs) are bound to their tokenizer, which maps raw text to a sequence of vocabulary items (tokens). This restricts their flexibility: for example, LMs trained primarily on English may still perform well in other natural and programming languages, but have vastly decreased efficiency due to their English-centric tokenizer. To mitigate this, we should be able to swap the original LM tokenizer with an arbitrary one, on the fly, without degrading performance. Hence, in this work we define a new problem: Zero-Shot Tokenizer Transfer (ZeTT). The challenge at the core of ZeTT is finding embeddings for the tokens in the vocabulary of the new tokenizer. Since prior heuristics for initializing embeddings often perform at chance level in a ZeTT setting, we propose a new solution: we train a hypernetwork taking a tokenizer as input and predicting the corresponding embeddings. We empirically demonstrate that the hypernetwork generalizes to new tokenizers both with encoder (e.g., XLM-R) and decoder LLMs (e.g., Mistral-7B). Our method comes close to the original models' performance in cross-lingual and coding tasks while markedly reducing the length of the tokenized sequence. We also find that the remaining gap can be quickly closed by continued training on less than 1B tokens. Finally, we show that a ZeTT hypernetwork trained for a base (L)LM can also be applied to fine-tuned variants without extra training. Overall, our results make substantial strides toward detaching LMs from their tokenizer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
