arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.03543cs.CLcs.AI
收藏

Tucano 2 Cool: 更好的开源葡萄牙语大语言模型

Tucano 2 Cool: Better Open Source LLMs for Portuguese

Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab(波恩-亚琛国际信息科技中心(b-it)/ CAISA实验室);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔机器学习与人工智能研究所);Center for Science and Thought(科学与思想研究中心);Helmholtz-Institut für Strahlen- und Kernphysik(亥姆霍兹辐射与核物理研究所);Bonn Sustainable AI Lab(波恩可持续AI实验室)

Nicholas Kluge Corrêa, Aniket Sen, Shiza Fatimah, Sophia Falk, Lennard Landgraf, Julia Kastner, Lucie Flek

AI总结:

Tucano 2推出了一系列开源葡萄牙语大语言模型,通过扩展数据集和改进训练方法,实现了在多种语言任务上的最佳性能。

摘要:

我们介绍了Tucano 2,一个完全开源的大语言模型(LLM)套件,具有0.5-3.7十亿参数,旨在解决葡萄牙语LLM开源开发中的某些空白。在我们之前的工作基础上,我们现在将我们的数据集GigaVerbo-v2扩展到新的质量和规模程度,同时引入了一个新的合成数据集GigaVerbo-v2 Synth,旨在填补GigaVerbo-v2中的缺失空白,并两个后训练数据集GigaVerbo-v2 SFT和GigaVerbo-v2 Preferences,使葡萄牙语LLM能够在检索增强生成、编程、工具使用、推理链等许多感兴趣的领域进行训练。通过广泛的消融研究,我们为Tucano 2套件(Base、Instruct和Think)设计了预训练和持续预训练配方,这些配方在几种葡萄牙语语言建模基准上实现了最先进的性能。我们还扩展和改进了我们先前工作中引入的评估工具包,从而得到一个全面的评估套件,它在不同的预训练、持续预训练和后训练制度中提供强信号。与Tucano 2相关的所有艺术品都公开发布,包括训练配方、日志和源代码,确保我们的工作可重复、可访问且可扩展,供更广泛的葡萄牙自然语言处理社区使用。

英文摘要:

We present Tucano 2, a fully open suite of large language models (LLMs) with 0.5-3.7 billion parameters, designed to address certain gaps in open-source development for Portuguese LLMs. Following our previous works, we now extend our dataset, GigaVerbo-v2, to a new degree of quality and scale, while also introducing a new synthetic dataset, GigaVerbo-v2 Synth, aimed at filling missing gaps in GigaVerbo-v2, and two post-training datasets, GigaVerbo-v2 SFT and GigaVerbo-v2 Preferences, that allow Portuguese LLMs to be trained in domains like retrieval augmented generation, coding, tool use, chain-of-thought reasoning, and many other domains of interest. Through extensive ablation studies, we design both pretraining and continual pretraining recipes for the Tucano 2 suite (Base, Instruct, and Think), which achieve state-of-the-art performance on several Portuguese-language modeling benchmarks. We also extend and refine the evaluation harness introduced in our earlier work, yielding a comprehensive evaluation suite that provides strong signals across different pretraining, continual pretraining, and post-training regimes. All artifacts associated with Tucano 2 are openly released, including training recipes, logs, and source code, ensuring that our work is reproducible, accessible, and extendable by the broader Portuguese NLP community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑