梵语是最高效的token语言吗？使用GPT、Gemini和SentencePiece的定量研究
Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece
AI总结：
本研究通过GPT、Gemini和SentencePiece验证梵语在token效率上的优势，发现其在无偏分词下比英语/印地语更紧凑，但最新模型仍无法完全捕捉其紧凑性。
摘要：
token是大型语言模型（LLMs）的基本单位。LLMs依赖于分词器将文本分割成这些token，而分词是决定计算和推理成本的主要因素。梵语是世界上最古老的语言之一，假设由于其形态学和语法规则，它能以更少的token表达更多的意义；然而，此前没有工作对其进行量化。我们使用包含701个平行诗节的Bhagavad Gita数据集，其中包括三种语言——梵语、英语和印地语，以及梵语转写成英语。我们测试了包括SentencePiece（SPM）、旧版GPT模型以及Gemini和GPT的最新一代分词器。我们使用token数量、每个token的字符数（token效率）和每个字符的token数（token成本）作为指标。结果显示，在无偏的SPM基准下，梵语与英语/印地语的token数量相差约2倍。梵语注释的英语/印地语翻译导致token数量增加了约20倍。GPT o200k base（最新版，用于GPT-4o）和Gemini（最新版）相比GPT cl100k base（用于GPT-4）显著减少了偏差，但仍未完全捕捉到梵语的紧凑性。这很重要，因为可能存在对非英语用户的惩罚偏差，这会增加token数量。这项研究为改进未来的分词器设计提供了基础，并展示了梵语在高度紧凑编码中的潜力，可以节省成本，同时加快训练和推理速度。代码和数据集可在https://github.com/anshulkr713/sanskrit-token-efficiency获取。
英文摘要：
Tokens are the basic units of Large Language Models (LLMs). LLMs rely on tokenizers to segment text into these tokens, and tokenization is the primary determinant of computational and inference cost. Sanskrit, one of the oldest languages, is hypothesized to express more meaning per token due to its morphology and grammar rules; however, no prior work has quantified this. We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi along with transliteration of Sanskrit into English. We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT. We use metrics of token count, characters per token (token efficiency), and tokens per character (token cost). Results show a ~2x difference in token counts between Sanskrit and English/Hindi under the unbiased SPM baseline. English/Hindi translations of Sanskrit commentary resulted in an approximately 20x increase in token count. GPT o200k base (latest, used by GPT-4o) and Gemini (latest) reduce bias by a significant degree compared to GPT cl100k base (used until GPT-4), but still fail to fully capture Sanskrit's compactness. This matters because there might be a penalty bias for non-English users, which inflates the token count. This research provides a foundation for improving future tokenizer design and shows the potential of Sanskrit for highly compact encoding, saving on cost while speeding up training and inference. The code and dataset are available at https://github.com/anshulkr713/sanskrit-token-efficiency
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
