基于音节粘合的印尼语LLM分词方法:来自Gasing识字学习系统的研究
Syllabic Agglutinative Tokenizations for Indonesian LLM: A Study from Gasing Literacy Learning System
AI总结:
本文提出基于音节的印尼语分词方法,结合教学与计算优化,提升分词效率与语言模型性能。
摘要:
本文提出了一种基于音节的分词方法,用于印尼大型语言模型,受Gasing识字学习系统教学方法的启发。借鉴信息论原理,我们开发了一种分词框架,首先在音节边界处分割印尼文本,然后应用字节对编码,创建一个与语言形态音系结构相匹配的词汇表。我们的方法首先通过规则分割识别高频音节,然后构建一个包含3500个token的紧凑词汇表,该词汇表在保留有意义的语言单位的同时,通过字符级回退保持覆盖范围。在印尼维基百科和印尼文化数字图书馆(PDBI)的folklore语料库上的实证评估表明,与传统分词方法相比,基于音节的方法在Rényi效率上取得了显著改进:音节方法的Rényi效率为0.74,而预训练多语言分词器的Rényi效率为0.50-0.64,尽管使用了比GPT-2小一个数量级的词汇表。这些改进来自于该方法在音节单元内内部化字符级依赖关系的能力,从而减少对语言模型的计算负担,同时尊重印尼语的粘合形态学。我们称基于此原理构建的LLM为TOBA LLM(基于粘合的最优分词),人类识字教学与计算优化原则的结合为开发语言学指导的分词策略提供了一个有前途的范式,特别是对于形态丰富且在自然语言处理中代表性不足的语言。
英文摘要:
This paper presents a novel syllable-based tokenization approach for Indonesian large language models, inspired by the Gasing Literacy Learning System's pedagogical methodology. Drawing on information-theoretic principles, we develop a tokenization framework that segments Indonesian text at syllable boundaries before applying byte-pair encoding, creating a vocabulary that aligns with the language's morphophonological structure. Our approach first identifies high-frequency syllables through rule-based segmentation, then constructs a compact vocabulary of 3,500 tokens that preserves meaningful linguistic units while maintaining coverage through character-level fallback. Empirical evaluation on Indonesian Wikipedia and folklore corpora from Indonesian Culture Digital Library (PDBI) demonstrates substantial improvements over conventional tokenization methods: the syllable-based approach achieves Rényi efficiency of 0.74 compared to 0.50-0.64 for pretrained multilingual tokenizers, while maintaining higher average token lengths (3.67 characters versus 2.72 for GPT-2) despite using a vocabulary an order of magnitude smaller. These gains emerge from the method's ability to internalize character-level dependencies within syllable units, reducing the computational burden on language models while respecting Indonesian's agglutinative morphology. We call the LLM built upon this principle, TOBA LLM (Tokenisasi Optimum Berbasis Aglutinasi), the convergence of human literacy pedagogy with computational optimization principles offers a promising paradigm for developing linguistically-informed tokenization strategies, particularly for morphologically rich and underrepresented languages in natural language processing.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。