大规模最优土耳其子词策略：数据、词汇和形态交互的系统评估
Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
AI总结：
本文系统评估了土耳其语子词分词策略，通过数据、词汇和形态交互的深入分析，提出统一的评估框架，并开源了相关工具和模型，为形态丰富语言的分词研究提供指导。
摘要：
子词划分是神经语言模型在形态丰富的语言（如土耳其语）中至关重要的设计选择，其中生成性黏着对词汇效率和形态忠实性提出了挑战。先前研究探讨了分词器家族和词汇量，但通常（i）在不系统控制分词器训练语料的情况下改变词汇量，（ii）提供有限的内在诊断，（iii）评估下游任务的狭窄切片。我们提出了首个全面、原则性的土耳其子词分词研究；一个“子词清单”，联合变化词汇量和分词器训练语料量（数据和词汇耦合），比较多个分词器家族在匹配参数预算下的表现（WordPiece、形态级别和字符基线），并在语义（NLI、STS、情感分析、NER）、语法（POS、依存解析）和形态敏感探测器上进行评估。为解释分词器为何成功或失败，我们引入了一个形态感知的诊断工具包，超越粗略汇总，深入到边界层面的微/宏观F1，解耦词根原子性与表面边界命中，过/欠分割指数，字符/词编辑距离（CER/WER），延续率，以及前缀/后缀类型覆盖和词级原子性。我们的贡献有四点：（i）词汇-语料-成功三元组的系统研究；（ii）统一的、形态感知的评估框架，将内在诊断与外在结果联系起来；（iii）受控比较，识别字符级别和形态级别分词何时有效；（iv）开源发布评估代码、分词管道和模型。作为同类研究中的首项工作，这一“子词清单”为在MRLs中构建有效分词器提供可操作的指导，并建立了可重复的基础，为未来研究铺平道路。
英文摘要：
Tokenization is a pivotal design choice for neural language modeling in morphologically rich languages (MRLs) such as Turkish, where productive agglutination challenges both vocabulary efficiency and morphological fidelity. Prior studies have explored tokenizer families and vocabulary sizes but typically (i) vary vocabulary without systematically controlling the tokenizer's training corpus, (ii) provide limited intrinsic diagnostics, and (iii) evaluate a narrow slice of downstream tasks. We present the first comprehensive, principled study of Turkish subword tokenization; a "subwords manifest", that jointly varies vocabulary size and tokenizer training corpus size (data and vocabulary coupling), compares multiple tokenizer families under matched parameter budgets (WordPiece, morphology level, and character baselines), and evaluates across semantic (NLI, STS, sentiment analysis, NER), syntactic (POS, dependency parsing), and morphology-sensitive probes. To explain why tokenizers succeed or fail, we introduce a morphology-aware diagnostic toolkit that goes beyond coarse aggregates to boundary-level micro/macro F1, decoupled lemma atomicity vs. surface boundary hits, over/under-segmentation indices, character/word edit distances (CER/WER), continuation rates, and affix-type coverage and token-level atomicity. Our contributions are fourfold: (i) a systematic investigation of the vocabulary-corpus-success triad; (ii) a unified, morphology-aware evaluation framework linking intrinsic diagnostics to extrinsic outcomes; (iii) controlled comparisons identifying when character-level and morphology-level tokenization pay off; and (iv) an open-source release of evaluation code, tokenizer pipelines, and models. As the first work of its kind, this "subwords manifest" delivers actionable guidance for building effective tokenizers in MRLs and establishes a reproducible foundation for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
