arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.12746cs.CL
收藏

在污染最小的大学入学考试通用知识任务上对语言模型的双语评估

Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination

UNED(国立远程教育大学);UNED Research Center in Natural Language Processing and Information Retrieval(国立远程教育大学自然语言处理与信息检索研究中心)
Eva Sánchez Salido, Roser Morante, Julio Gonzalo, Guillermo Marco, Jorge Carrillo-de-Albornoz, Laura Plaza, Enrique Amigó, Andrés Fernández, Alejandro Benito-Sa… 展开作者

Eva Sánchez Salido, Roser Morante, Julio Gonzalo, Guillermo Marco, Jorge Carrillo-de-Albornoz, Laura Plaza, Enrique Amigó, Andrés Fernández, Alejandro Benito-Santos, Adrián Ghajari Espinosa, Victor Fresno

AI总结:

该研究构建了含1003道题的双语大学入学考试数据集UNED-ACCESS 2024,在零样本设置下评估多款语言模型,发现小模型双语性能差距最高达37%,且该数据集排名与MMLU高度相关,可有效衡量学科性能。

摘要:

本文提出UNED-ACCESS 2024,这是一个包含1003道大学入学水平考试选择题的双语数据集,涵盖西班牙语和英语两种语言。题目最初以西班牙语命制,后经人工翻译为英语,且从未被公开发布过。研究选取了当前多款开源模型和专有模型,在统一的零样本实验设置下,分别在UNED-ACCESS 2024数据集和MMLU(大规模多任务语言理解)题目的等效子集上开展评估。结果显示:(1)推理类题目对模型而言具有挑战性;(2)小模型表现差于大模型,且其在西班牙语上的性能衰减速度快于英语;(3)最优模型在两种语言上的性能差距可忽略不计,而小模型的该差距最高可达37%。模型在UNED-ACCESS 2024上的排名在英语和西班牙语中几乎一致,且与MMLU上的排名具有高度相关性(皮尔逊相关系数0.98),表明该小型数据集具备足够的多样性与代表性,可用于衡量不同学科的模型性能。

英文摘要:

In this article we present UNED-ACCESS 2024, a bilingual dataset that consists of 1003 multiple-choice questions of university entrance level exams in Spanish and English. Questions are originally formulated in Spanish and translated manually into English, and have not ever been publicly released. A selection of current open-source and proprietary models are evaluated in a uniform zero-shot experimental setting both on the UNED-ACCESS 2024 dataset and on an equivalent subset of MMLU questions. Results show that (i) reasoning questions are challenging for models, (ii) smaller models perform worse than larger models and degrade faster in Spanish than in English and (iii) the performance gap between languages is negligible for the best models and grows up to 37% for smaller models. Model ranking on UNED-ACCESS 2024 is almost identical in English and Spanish, and has also a high correlation (0.98 Pearson) with ranking on MMLU, suggesting that a small dataset is sufficiently diverse and representative to measure performance by discipline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑