We introduce Sabiá-2, a family of large language models trained on Portuguese texts. The models are evaluated on a diverse range of exams, including entry-level tests for Brazilian universities, professional certification exams, and graduate-level exams for various disciplines such as accounting, economics, engineering, law and medicine. Our results reveal that our best model so far, Sabiá-2 Medium, matches or surpasses GPT-4's performance in 23 out of 64 exams and outperforms GPT-3.5 in 58 out of 64 exams. Notably, specialization has a significant impact on a model's performance without the need to increase its size, allowing us to offer Sabiá-2 Medium at a price per token that is 10 times cheaper than GPT-4. Finally, we identified that math and coding are key abilities that need improvement.
Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs
Institute of Science and Technology(科学技术研究院);Federal University of São Paulo (UNIFESP)(圣保罗联邦大学)
Tucano 2 Cool: Better Open Source LLMs for Portuguese
Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab(波恩-亚琛国际信息科技中心(b-it)/ CAISA实验室);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔机器学习与人工智能研究所);Center for Science and Thought(科学与思想研究中心);Helmholtz-Institut für Strahlen- und Kernphysik(亥姆霍兹辐射与核物理研究所);Bonn Sustainable AI Lab(波恩可持续AI实验室)
MOPrompt: Multi-objective Semantic Evolution for Prompt Optimization
Computing Department, Universidade Federal de Ouro Preto(联邦大学奥鲁普里托 computing 部门);Federal Institute of Minas Gerais(巴西矿业高等学院);Postgraduate Program in Computer Science, Federal University of Ouro Preto(联邦大学奥鲁普里托 计算科学研究生项目)
AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
NOVA School of Science and Technology(新大学科学与技术学院);NOVA LINCS(新大学LINCS实验室);Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等技术学院);Fundação para a Ciência e Tecnologia(科学技术基金会)
Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge
National Laboratory for Scientific Computing (LNCC)(国家科学计算实验室);Institute of Artificial Intelligence(人工智能研究所)
Multilingual Natural Language Processing Model for Radiology Reports -- The Summary is all you need!
ALGORITMI Research Centre/LASI, University of Minho(米尼奥大学 ALGORITMI 研究中心/LASI);Institute for Artificial Intelligence in Medicine, University Medicine Essen (AöR)(埃森大学医学院人工智能医学研究所);Department of Diagnostic and Interventional Radiology and Neuroradiology, University Medicine Essen (AöR)(埃森大学医学院诊断与介入放射学及神经放射学系);Institute for Computer Graphics and Vision, Graz University of Technology(格拉茨技术大学计算机图形与视觉研究所)
From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns
Faculdade de Engenharia, Universidade do Porto(工程学院,葡萄牙波尔图大学);Departamento de Investigação e Tecnologia(研究与技术部门);Divisão de Tecnologias Educativas(教育技术部门);Escola Básica Mirante dos Sonhos(梦想之镜初级学校)
Tailored untruths: How personalisation challenges LLM safeguards
University of Sheffield(谢菲尔德大学);University of Copenhagen(哥本哈根大学);Big Data for Smart Society Institute (GATE)(大数据智能社会研究所(GATE));University of São Paulo(圣保罗大学)
RDF2PT: Generating Brazilian Portuguese Texts from RDF Data
University of Leipzig(莱比锡大学);Tilburg University(蒂尔堡大学);University of Wolverhampton(伍尔弗汉普顿大学);Military Institute of Engineering - Brazil(巴西军事工程学院);Federal University of Rio de Janeiro - Brazil(巴西里约热内卢联邦大学);German Federal Institute for Risk Assessment - Germany(德国联邦风险评估研究所);Paderborn University(帕德博恩大学)