SampoNLP：一种用于子词分词形态分析的自指工具包
SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
AI总结：
SampoNLP通过自指原子性评分创建形态词典，评估BPE分词器在不同词汇量下的性能，提出IPS指标并提供最优词汇量推荐。
摘要：
子词分词的质量对大语言模型至关重要，然而评估具有丰富形态的乌拉尔语系语言的分词器却受到缺乏干净的词素词典的阻碍。我们介绍了SampoNLP，一种无需语料库的工具包，利用MDL启发的自指原子性评分来创建形态词典，通过内部结构线索过滤复合形式，适用于低资源环境。使用SampoNLP为芬兰语、匈牙利语和爱沙尼亚语生成的高纯度词典，我们对BPE分词器在不同词汇量（8k-256k）范围内进行了系统评估。我们提出了一种统一的指标，即综合性能评分（IPS），以在词素覆盖和过度分割之间取得平衡。通过分析IPS曲线，我们确定了“拐点”即边际收益递减点，并为这些语言提供了首次基于实证的最优词汇量（k）的推荐。我们的研究不仅提供了实用指导，还定量展示了标准BPE在高度黏着语言中的局限性。SampoNLP库和所有生成的资源均已公开：https://github.com/AragonerUA/SampoNLP
英文摘要：
The quality of subword tokenization is critical for Large Language Models, yet evaluating tokenizers for morphologically rich Uralic languages is hampered by the lack of clean morpheme lexicons. We introduce SampoNLP, a corpus-free toolkit for morphological lexicon creation using MDL-inspired Self-Referential Atomicity Scoring, which filters composite forms through internal structural cues - suited for low-resource settings. Using the high-purity lexicons generated by SampoNLP for Finnish, Hungarian, and Estonian, we conduct a systematic evaluation of BPE tokenizers across a range of vocabulary sizes (8k-256k). We propose a unified metric, the Integrated Performance Score (IPS), to navigate the trade-off between morpheme coverage and over-splitting. By analyzing the IPS curves, we identify the "elbow points" of diminishing returns and provide the first empirically grounded recommendations for optimal vocabulary sizes (k) in these languages. Our study not only offers practical guidance but also quantitatively demonstrates the limitations of standard BPE for highly agglutinative languages. The SampoNLP library and all generated resources are made publicly available: https://github.com/AragonerUA/SampoNLP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
