GlórIA——一个面向葡萄牙语的生成式开放大语言模型
GlórIA -- A Generative and Open Large Language Model for Portuguese
AI总结：
本文提出欧洲葡萄牙语大语言模型GlórIA，基于350亿token的PT-PT语料库预训练，并引入首个葡萄牙语零样本语言建模基准CALAME-PT，在语言建模和下游任务上显著优于现有开放模型。
摘要：
自然语言任务已取得重大进展，这在很大程度上归功于强大的大语言模型（LLMs）的出现。这些模型在海量且多样化的语料库上进行预训练，已越来越能够理解语言的复杂性。尽管针对许多高资源语言已有大量LLMs，但面向欧洲葡萄牙语的此类模型仍然有限。我们提出GlórIA，一个稳健的欧洲葡萄牙语解码器LLM。为了预训练GlórIA，我们汇集了一个包含350亿个token的综合性PT-PT文本语料库，来源多样。我们介绍了预训练方法，随后评估了该模型在多个下游任务上的有效性。此外，为了评估我们模型的语言建模能力，我们引入CALAME-PT（面向葡萄牙语的上下文感知语言建模评估），这是首个葡萄牙语零样本语言建模基准。评估表明，GlórIA在语言建模上显著优于现有的开放PT解码器模型，并且能够生成合理、知识丰富且连贯的PT-PT文本。该模型还展现出在各种下游任务上的强大潜力。
英文摘要：
Significant strides have been made in natural language tasks, largely attributed to the emergence of powerful large language models (LLMs). These models, pre-trained on extensive and diverse corpora, have become increasingly capable of comprehending the intricacies of language. Despite the abundance of LLMs for many high-resource languages, the availability of such models remains limited for European Portuguese. We introduce GlórIA, a robust European Portuguese decoder LLM. To pre-train GlórIA, we assembled a comprehensive PT-PT text corpus comprising 35 billion tokens from various sources. We present our pre-training methodology, followed by an assessment of the model's effectiveness on multiple downstream tasks. Additionally, to evaluate our models' language modeling capabilities, we introduce CALAME-PT (Context-Aware LAnguage Modeling Evaluation for Portuguese), the first Portuguese zero-shot language-modeling benchmark. Evaluation shows that GlórIA significantly outperforms existing open PT decoder models in language modeling and that it can generate sound, knowledge-rich, and coherent PT-PT text. The model also exhibits strong potential for various downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
