语言完整性的分词标准：以土耳其语为基准
Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark
AI总结：
本文提出一个以土耳其语为基准的分词评估框架，通过词汇量、token数量、处理时间、%TR和token纯度五个指标系统评估分词器，发现%TR与下游性能相关性更强，且更大模型参数并不必然带来更好分词质量，为形态复杂和低资源语言的分词优化提供了新标准。
摘要：
分词是自然语言处理（NLP）中的基础预处理步骤，直接影响大语言模型（LLMs）捕捉句法、形态句法和语义结构的能力。本文提出了一种系统评估分词策略的新框架，以应对形态丰富和低资源语言中的挑战。该框架使用来自大规模多任务语言理解（MMLU）基准的6,200道土耳其语多项选择题数据集，从五个关键指标评估分词器：词汇表大小、token数量、处理时间、目标语言token百分比（%TR）和token纯度。这些指标为评估分词器在多大程度上保留语言结构提供了一种结构化方法。%TR衡量目标语言中有效词的比例，而%Pure评估token与有意义语言单位（如词根和有效语素）的对齐程度，从而最小化语义碎片化。研究结果表明，作为一项关键指标引入的%TR与下游性能（例如MMLU分数）的相关性强于token纯度，凸显了其在提高模型准确性方面的作用。此外，更大的模型参数量并不一定能带来更好的分词质量或更好的结果，这强调了优先考虑语言对齐的定制化分词策略的重要性。该框架为开发针对形态复杂和低资源语言优化的稳健分词方法设立了新标准。未来工作将改进形态分析、探索领域特定定制，并进行跨语言评估，以进一步提升分词实践。
英文摘要：
Tokenization is a fundamental preprocessing step in NLP, directly impacting large language models' (LLMs) ability to capture syntactic, morphosyntactic, and semantic structures. This paper introduces a novel framework for systematically evaluating tokenization strategies, addressing challenges in morphologically rich and low-resource languages. Using a Turkish dataset of 6,200 multiple-choice questions from the Massive Multitask Language Understanding (MMLU) benchmark, the framework assesses tokenizers across five key metrics: vocabulary size, token count, processing time, language-specific token percentages (\%TR), and token purity. These metrics provide a structured approach to evaluating how well tokenizers preserve linguistic structures. While \%TR measures the proportion of valid words in the target language, \%Pure assesses the alignment of tokens with meaningful linguistic units, such as roots and valid morphemes, minimizing semantic fragmentation. The findings reveal that \%TR, introduced as a critical metric, exhibits a stronger correlation with downstream performance (e.g., MMLU scores) than token purity, emphasizing its role in improving model accuracy. Additionally, larger model parameters do not necessarily yield better tokenization quality or enhanced results, highlighting the importance of tailored tokenization strategies that prioritize linguistic alignment. This framework sets a new standard for developing robust tokenization methods optimized for morphologically complex and low-resource languages. Future work will refine morphological analysis, explore domain-specific customizations, and conduct cross-linguistic evaluations to further enhance tokenization practices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
