超越字面标记重叠：多语言性的标记可对齐性
Beyond Literal Token Overlap: Token Alignability for Multilinguality
AI总结：
针对以往标记重叠指标对不同文字语言对解释力有限的问题，本文提出子词标记可对齐性指标，在不同文字场景下更好预测多语言性，可用于识别跨语言迁移最佳语言对并指导多语言标记器构建。
摘要：
以往研究将标记重叠甚至标记分布的相似性视为语言模型中多语言性和跨语言知识迁移的预测指标。然而，这些非常字面的指标会给使用不同文字的语言对分配较大距离，尽管这些语言对可能表现出良好的跨语言性。这限制了标记重叠对使用不同文字或遵循不同正字法惯例的语言对之间知识迁移的解释力。在本文中，我们提出子词标记可对齐性作为理解多语言标记化影响和质量的新方法。特别是，当文字差异较大且字面标记重叠度较低时，该指标能更好地预测多语言性。我们在编码器和解码器模型的背景下分析了这一指标，研究了数据规模作为潜在干扰因素的影响，并讨论了这一见解如何应用于未来的多语言标记化。我们推荐我们的子词标记可对齐性指标，用于识别跨语言迁移的最佳语言对，并指导未来构建更好的多语言标记器。我们发布了代码和可复现性细节。
英文摘要：
Previous work has considered token overlap, or even similarity of token distributions, as predictors for multilinguality and cross-lingual knowledge transfer in language models. However, these very literal metrics assign large distances to language pairs with different scripts, which can nevertheless show good cross-linguality. This limits the explanatory strength of token overlap for knowledge transfer between language pairs that use distinct scripts or follow different orthographic conventions. In this paper, we propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation. In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. We analyse this metric in the context of both encoder and decoder models, look at data size as a potential distractor, and discuss how this insight may be applied to multilingual tokenisation in future work. We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer, as well as to guide the construction of better multilingual tokenisers in the future. We publish our code and reproducibility details.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
