Paramanu：紧凑且具有竞争力的单语语言模型，用于低资源的形态丰富的印度语言
Paramanu: Compact and Competitive Monolingual Language Models for Low-Resource Morphologically Rich Indian Languages
AI总结：
Paramanu通过低成本训练方法，在五种印度语言上实现了高性能，优于多数多语言模型。
摘要：
多语言大型语言模型（LLMs）在预训练过程中成本高昂，且常常在不同语言和数据集之间存在不平衡，存在以英语为中心的偏见，以及在形态丰富但资源有限的语言中，分词过度分割的问题，以及多语言的诅咒。我们介绍了PARAMANU，这是首个专为印度语言设计的自回归语言模型家族，这些模型是基于开源语言特定数据从头开始训练的，针对五种最广泛使用的印度语言：孟加拉语、印地语、马拉地语、泰米尔语和泰卢固语。所有模型都设计用于经济性，并在单块GPU上以不到1000美元的预算进行训练，使资源有限的研究人员能够构建具有竞争力的语言模型。为了解决资源有限的挑战，我们开发了形态对齐、低生育率的分词器，提出了基于插值的方法来计算RoPE基于的token位置索引，以更高效地训练更长的序列。我们还创建了在孟加拉语中翻译到其他四种语言的指令微调数据集。尽管它们的规模较小（10800万-36700万参数），Paramanu在性能-效率权衡上表现出色，并在所有五种语言中优于大多数更大的多语言模型。我们的集合可在https://huggingface.co/collections/mitodru/paramanu上获取。
英文摘要：
Multilingual large language models (LLMs) are expensive to pretrain and often suffer from imbalances across languages and datasets, English-centric bias, tokenizer oversegmentation for morphologically rich low-resource languages, and the curse of multilinguality. We introduce PARAMANU, the first family of Indian-only autoregressive language models trained from scratch on open-source language-specific data for the five most spoken Indian languages: Bengali, Hindi, Marathi, Tamil, and Telugu. All models are designed for affordability and are trained on a single GPU with a budget under $1,000, allowing under-resourced researchers to build competitive language models. To address low-resource challenges, we develop morphology-aligned, low-fertility tokenizers, propose an interpolation-based method for token position indices in RoPE based scaling to train longer sequences efficiently. We also create instruction-tuning datasets in Bangla that are translated to the other four languages. Despite their small size (108M-367M parameters), Paramanu achieves a strong performance-efficiency tradeoff and outperforms most larger multilingual models across all five languages. Our collection is available at https://huggingface.co/collections/mitodru/paramanu.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
