语言模型中的平等语言表示：一切始于分词器
Egalitarian Language Representation in Language Models: It All Begins with Tokenizers
AI总结：
本文针对复杂文字语言（如泰米尔语、僧伽罗语、印地语）在分词中的不公平表示问题，提出改进的GPE算法，证明预分词比分词算法更关键，且字素提取优于字节级分词器。
摘要：
分词器在人类语言与语言模型的潜在空间之间架起桥梁，影响语言在这些模型中的表示方式。由于以英语为中心的大型语言模型（LLM）的广泛流行，人们正在努力将其适应于其他语言。然而，我们证明，从分词的角度来看，并非所有分词器都能为泰米尔语、僧伽罗语和印地语等复杂文字语言提供公平的表示，这主要归因于预分词方法的选择。我们进一步表明，在实现这些复杂文字语言的平等表示方面，预分词比分词算法本身起着更为关键的作用。为了解决这一问题，我们对字节对编码（BPE）算法进行了改进，引入字素，称之为字素对编码（GPE）。我们的实验表明，对于复杂文字，基于字素的字符提取优于字节级分词器。我们通过在泰米尔语、僧伽罗语和印地语上的实验验证了这种方法。
英文摘要：
Tokenizers act as a bridge between human language and the latent space of language models, influencing how language is represented in these models. Due to the immense popularity of English-Centric Large Language Models (LLMs), efforts are being made to adapt them for other languages. However, we demonstrate that, from a tokenization standpoint, not all tokenizers offer fair representation for complex script languages such as Tamil, Sinhala, and Hindi, primarily due to the choice of pre-tokenization methods. We go further to show that pre-tokenization plays a more critical role than the tokenization algorithm itself in achieving an egalitarian representation of these complex script languages. To address this, we introduce an improvement to the Byte Pair Encoding (BPE) algorithm by incorporating graphemes, which we term Grapheme Pair Encoding (GPE). Our experiments show that grapheme-based character extraction outperforms byte-level tokenizers for complex scripts. We validate this approach through experiments on Tamil, Sinhala, and Hindi.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
