arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.16640cs.CLcs.LG
收藏

TeenyTinyLlama:用巴西葡萄牙语训练的开源小型语言模型

TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese

University of Bonn(波恩大学);Pontifical Catholic University of Rio Grande do Sul(南里奥格兰德天主教大学)

Nicholas Kluge Corrêa, Sophia Falk, Shiza Fatimah, Aniket Sen, Nythamar de Oliveira

AI总结:

针对大语言模型在低资源语言上表现不佳、多语言模型存在计算与许可限制的问题,研究推出两款巴西葡萄牙语紧凑型生成模型TeenyTinyLlama,以Apache 2.0许可证开源供社区使用。

摘要:

大语言模型(LLM)已显著推动自然语言处理发展,但各语言的进展并不均衡。多数LLM以英语等高资源语言训练,多语言模型的表现通常不及单语言模型。此外,其多语言基础的特性有时会限制衍生成果,比如计算需求与许可制度。本研究记录了专为低资源场景定制的开源基础模型的开发过程、局限性与优势,即TeenyTinyLlama模型对:两款用于巴西葡萄牙语文本生成的紧凑型模型。我们以宽松的Apache 2.0许可证在GitHub和Hugging Face上发布,供社区使用与进一步开发,详见https://github.com/Nkluge-correa/TeenyTinyLlama。

英文摘要:

Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. In this study, we document the development of open-foundation models tailored for use in low-resource settings, their limitations, and their benefits. This is the TeenyTinyLlama pair: two compact models for Brazilian Portuguese text generation. We release them under the permissive Apache 2.0 license on GitHub and Hugging Face for community use and further development. See https://github.com/Nkluge-correa/TeenyTinyLlama

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑