具有意义的标记：一种针对土耳其语的混合标记方法
Tokens with Meaning: A Hybrid Tokenization Approach for Turkish
AI总结：
本文提出了一种结合词典驱动形态分割、音学规范化和受控子词回退的土耳其语混合标记器，通过TR-MMLU数据集评估，其在形态和词汇单元匹配度上优于现有方法。
摘要：
标记化影响语言模型对形态和意义在NLP中的感知方式，但广泛使用的基于频率的子词标记器（如字节对编码和词片）可能在形态丰富和黏着性语言中导致碎片化，从而模糊形态边界。我们介绍了一种受语言学启发的混合标记器，用于土耳其语，结合（i）词典驱动的形态分割（根和词缀）、（ii）音学规范化将所有变体映射到共享标识符，以及（iii）受控的子词回退以覆盖词汇外的覆盖。具体而言，我们发布的土耳其词汇包含22,231个根标记映射到20,000个标准根标识符（带有前导空格以标记词边界），72个词缀标识符覆盖177个所有变体表层形式，以及12,696个子词单元；一个正字法情况标记保留大写而不扩大词汇量。我们在TR-MMLU数据集上评估标记化质量，使用两种语言学对齐度量：土耳其标记百分比（TR%），即所生成的标记中对应土耳其词汇/形态单元的比例，以及纯标记百分比（Pure%），即与无歧义根/词缀边界对齐的标记比例。所提出的标记器在TR-MMLU上达到90.29% TR%和85.80% Pure%，显著超过几种通用标记器。我们进一步通过下游句子嵌入基准测试验证实际用途，在严格随机初始化控制下隔离标记器归纳偏置。在四个匹配模型（土耳其Tokenizer、CosmosGPT2、Mursit和Tabi）中，土耳其Tokenizer在土耳其STS基准测试中优于所有基线，在MTEB-TR上达到最强的整体平均。它还在TurBLiMP下以基于质心的代理实现了最强的平均准确率。
英文摘要：
Tokenization shapes how language models perceive morphology and meaning in NLP, yet widely used frequency-driven subword tokenizers (e.g., Byte Pair Encoding and WordPiece) can fragment morphologically rich and agglutinative languages in ways that obscure morpheme boundaries. We introduce a linguistically informed hybrid tokenizer for Turkish that combines (i) dictionary-driven morphological segmentation (roots and affixes), (ii) phonological normalization that maps allomorphic variants to shared identifiers, and (iii) a controlled subword fallback for out-of-vocabulary coverage. Concretely, our released Turkish vocabulary contains 22,231 root tokens mapped to 20,000 canonical root identifiers (with leading spaces to mark word boundaries), 72 affix identifiers that cover 177 allomorphic surface forms, and 12,696 subword units; an orthographic case token preserves capitalization without inflating the vocabulary. We evaluate tokenization quality on the TR-MMLU dataset using two linguistic alignment metrics: Turkish Token Percentage (TR~\%), the proportion of produced tokens that correspond to Turkish lexical/morphemic units under our lexical resources, and Pure Token Percentage (Pure~\%), the proportion of tokens aligning with unambiguous root/affix boundaries. The proposed tokenizer reaches 90.29\% TR~\% and 85.80\% Pure~\% on TR-MMLU, substantially exceeding several general-purpose tokenizers. We further validate practical utility with downstream sentence embedding benchmarks under a strict \emph{random initialization} control to isolate tokenizer inductive bias. Across four matched models (TurkishTokenizer, CosmosGPT2, Mursit, and Tabi), TurkishTokenizer outperforms all baselines on the Turkish STS Benchmark and achieves the strongest overall average on MTEB-TR. It also yields the strongest average accuracy on the TurBLiMP under a centroid-based proxy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
