Tucano 2 Cool: 更好的开源葡萄牙语大语言模型
Tucano 2 Cool: Better Open Source LLMs for Portuguese
AI总结:
Tucano 2推出了一系列开源葡萄牙语大语言模型,通过扩展数据集和改进训练方法,实现了在多种语言任务上的最佳性能。
摘要:
我们介绍了Tucano 2,一个完全开源的大语言模型(LLM)套件,具有0.5-3.7十亿参数,旨在解决葡萄牙语LLM开源开发中的某些空白。在我们之前的工作基础上,我们现在将我们的数据集GigaVerbo-v2扩展到新的质量和规模程度,同时引入了一个新的合成数据集GigaVerbo-v2 Synth,旨在填补GigaVerbo-v2中的缺失空白,并两个后训练数据集GigaVerbo-v2 SFT和GigaVerbo-v2 Preferences,使葡萄牙语LLM能够在检索增强生成、编程、工具使用、推理链等许多感兴趣的领域进行训练。通过广泛的消融研究,我们为Tucano 2套件(Base、Instruct和Think)设计了预训练和持续预训练配方,这些配方在几种葡萄牙语语言建模基准上实现了最先进的性能。我们还扩展和改进了我们先前工作中引入的评估工具包,从而得到一个全面的评估套件,它在不同的预训练、持续预训练和后训练制度中提供强信号。与Tucano 2相关的所有艺术品都公开发布,包括训练配方、日志和源代码,确保我们的工作可重复、可访问且可扩展,供更广泛的葡萄牙自然语言处理社区使用。
英文摘要:
We present Tucano 2, a fully open suite of large language models (LLMs) with 0.5-3.7 billion parameters, designed to address certain gaps in open-source development for Portuguese LLMs. Following our previous works, we now extend our dataset, GigaVerbo-v2, to a new degree of quality and scale, while also introducing a new synthetic dataset, GigaVerbo-v2 Synth, aimed at filling missing gaps in GigaVerbo-v2, and two post-training datasets, GigaVerbo-v2 SFT and GigaVerbo-v2 Preferences, that allow Portuguese LLMs to be trained in domains like retrieval augmented generation, coding, tool use, chain-of-thought reasoning, and many other domains of interest. Through extensive ablation studies, we design both pretraining and continual pretraining recipes for the Tucano 2 suite (Base, Instruct, and Think), which achieve state-of-the-art performance on several Portuguese-language modeling benchmarks. We also extend and refine the evaluation harness introduced in our earlier work, yielding a comprehensive evaluation suite that provides strong signals across different pretraining, continual pretraining, and post-training regimes. All artifacts associated with Tucano 2 are openly released, including training recipes, logs, and source code, ensuring that our work is reproducible, accessible, and extendable by the broader Portuguese NLP community.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。