分词至关重要：提升印度语言零样本命名实体识别
Tokenization Matters: Improving Zero-Shot NER for Indic Languages
AI总结：
本研究系统比较了BPE、SentencePiece和字符级分词策略，发现SentencePiece在低资源印度语言的零样本NER中优于BPE，因其更好地保留实体一致性和语言结构，尤其适用于形态丰富的极低资源语言。
摘要：
分词是自然语言处理（NLP）中的关键组成部分，尤其对于低资源语言而言，子词切分会影响词汇结构及下游任务准确性。尽管字节对编码（BPE）是多语言模型中的标准分词方法，但由于其在处理形态复杂性方面的局限性，其对于低资源印度语言命名实体识别（NER）的适用性仍未得到充分探索。在本工作中，我们使用IndicBERT系统比较了BPE、SentencePiece和字符级分词策略，用于阿萨姆语、孟加拉语、马拉地语和奥里亚语等低资源印度语言以及桑塔利语、曼尼普尔语和信德语等极低资源印度语言的NER任务。我们评估了内在的语言学属性——分词效率、词汇外（OOV）率以及形态保留——以及外在的下游性能，包括微调和零样本跨语言迁移。我们的实验表明，在低资源印度语言的NER中，SentencePiece始终是比BPE更优的方法，尤其是在零样本跨语言场景下，因为它能更好地保持实体一致性。虽然BPE提供了最紧凑的分词形式，但它在未见语言上测试时会错误分类甚至无法识别实体标签，因此不具备泛化能力。相比之下，SentencePiece构成了更好的语言结构保留模型，有利于极低资源和形态丰富的印度语言（如桑塔利语和曼尼普尔语）实现卓越的实体识别，并能在不同文字（如用阿拉伯文书写的信德语）间实现高度泛化。结果表明，在多语言和低资源印度语NLP应用中，SentencePiece是NER更有效的分词策略。
英文摘要：
Tokenization is a critical component of Natural Language Processing (NLP), especially for low resource languages, where subword segmentation influences vocabulary structure and downstream task accuracy. Although Byte Pair Encoding (BPE) is a standard tokenization method in multilingual language models, its suitability for Named Entity Recognition (NER) in low resource Indic languages remains underexplored due to its limitations in handling morphological complexity. In this work, we systematically compare BPE, SentencePiece, and Character Level tokenization strategies using IndicBERT for NER tasks in low resource Indic languages like Assamese, Bengali, Marathi, and Odia, as well as extremely low resource Indic languages like Santali, Manipuri, and Sindhi. We assess both intrinsic linguistic properties tokenization efficiency, out of vocabulary (OOV) rates, and morphological preservation as well as extrinsic downstream performance, including fine tuning and zero shot cross lingual transfer. Our experiments show that SentencePiece is a consistently better performing approach than BPE for NER in low resource Indic Languages, particularly in zero shot cross lingual settings, as it better preserves entity consistency. While BPE provides the most compact tokenization form, it is not capable of generalization because it misclassifies or even fails to recognize entity labels when tested on unseen languages. In contrast, SentencePiece constitutes a better linguistic structural preservation model, benefiting extremely low resource and morphologically rich Indic languages, such as Santali and Manipuri, for superior entity recognition, as well as high generalization across scripts, such as Sindhi, written in Arabic. The results point to SentencePiece as the more effective tokenization strategy for NER within multilingual and low resource Indic NLP applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
