分词影响多语言语言建模：评估跨语言的词汇分配与重叠
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
AI总结：
本文通过提出评估子词分词器词汇表示与重叠的新准则，揭示词汇重叠对不同下游任务利弊不一，并为多语言模型预训练前的分词器选择提供指导。
摘要：
多语言语言模型近来作为一种在单一模型中表示多种语言的有前景的解决方案而受到关注。本文提出了新的准则，用于评估子词分词器中观察到的词汇表示质量和词汇重叠情况。我们的研究结果表明，跨语言的词汇重叠实际上可能对某些下游任务（词性标注、依存树标注）有害。相比之下，命名实体识别和句子级任务（跨语言检索、自然语言推理）则受益于共享词汇。我们还观察到，多语言词汇中语言特定标记的覆盖率显著影响词级任务。我们的研究提供了对分词器在多语言语言模型中作用的更深入理解，并为未来的模型开发者在进行昂贵的模型预训练之前，为其特定应用选择最合适的分词器提供了指导。
英文摘要：
Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model. In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling). In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary. We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the word-level tasks. Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
