73种语言的屈折变化:一个用于多语言屈折的小型单一模型
Flexing in 73 Languages: A Single Small Model for Multilingual Inflection
AI总结:
本文提出一个轻量级多语言屈折模型,在73种语言上联合训练,优于单语言基线,并引入频率加权重采样方法,开源代码,解决未见词处理问题。
摘要:
我们提出了一种紧凑的、单一模型的方法来解决多语言屈折变化任务,即从基础词元生成屈折词形以表达语法范畴。我们的模型在73种语言的数据上联合训练,轻量级、对未见词具有鲁棒性,并且在大多数语言中优于单语言基线。这证明了多语言建模在屈折变化任务中的有效性,并突出了其实用优势:通过消除管理和重新训练数十个单独的单语言模型的需求,简化了部署。除了标准的SIGMORPHON共享任务基准外,我们还在73个通用依赖(UD)树库上评估了我们的单语言和多语言模型,提取了词元-标签-词形三元组及其频率计数。为了确保现实的数据划分,我们引入了一种新颖的、频率加权且词元不相交的训练-开发-测试重采样程序。我们的工作解决了缺乏一个开源、通用、多语言形态屈折系统的问题,该系统能够处理包括捷克语在内的广泛语言中的未见词。所有代码已在https://github.com/tomsouri/multilingual-inflection公开发布。
英文摘要:
We present a compact, single-model approach to multilingual inflection, the task of generating inflected word forms from base lemmas to express grammatical categories. Our model, trained jointly on data from 73 languages, is lightweight, robust to unseen words, and outperforms monolingual baselines in most languages. This demonstrates the effectiveness of multilingual modeling for inflection and highlights its practical benefits: simplifying deployment by eliminating the need to manage and retrain dozens of separate monolingual models. In addition to the standard SIGMORPHON shared task benchmarks, we evaluate our monolingual and multilingual models on 73 Universal Dependencies (UD) treebanks, extracting lemma-tag-form triples and their frequency counts. To ensure realistic data splits, we introduce a novel frequency-weighted, lemma-disjoint train-dev-test resampling procedure. Our work addresses the lack of an open-source, general-purpose, multilingual morphological inflection system capable of handling unseen words across a wide range of languages, including Czech. All code is publicly released at: https://github.com/tomsouri/multilingual-inflection.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。