arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.18697cs.CL
收藏

Invalsi基准:衡量大语言模型的意大利语语言与数学理解能力

The Invalsi Benchmarks: measuring Linguistic and Mathematical understanding of Large Language Models in Italian

ISTI CNR(ISTI国家研究委员会)

Giovanni Puccetti, Maria Cassese, Andrea Esuli

AI总结:

针对意大利语本土LLM评估基准不足的问题,本研究构建3项不同难度的意大利语语言与数学理解基准,评估10个模型的表现并与学生平均水平对比,相关资源将在论文录用后开源。

摘要:

尽管意大利语属于高资源语言,但目前用于评估生成式大语言模型(LLM)意大利语能力的本土基准仍十分匮乏。本研究提出三项全新基准:用于评估模型意大利语数学理解能力的Invalsi MATE、用于评估意大利语语言理解能力的Invalsi ITA,以及面向更复杂数学理解任务的Olimpiadi MATE。\n前两项基准基于意大利教育体系中面向6至18岁学生开展的Invalsi测试构建,已得到多位教学与教育学专家的验证;第三项基准则源自意大利高中数学奥林匹克竞赛。\n我们在这些基准上评估了10个性能优异的语言模型,结果显示:模型在Invalsi MATE上的准确率上限为71%,由Llama 3.1 70b instruct达成;在Invalsi ITA上的准确率上限为88%。我们将LLM在Invalsi MATE与Invalsi ITA上的表现与意大利学生的平均水平进行对比,发现仅Llama 3.1在Invalsi MATE上的表现超过学生平均水平,而多数模型在Invalsi ITA上均能超过该水平。研究同时表明Olimpiadi MATE比Invalsi MATE更具挑战性,其最高准确率为45%,由Llama 3.1 405b instruct达成。\n论文录用后,我们将公开相关数据与评估代码。

英文摘要:

While Italian is a high-resource language, there are few Italian-native benchmarks to evaluate generative Large Language Models (LLMs) in this language. This work presents three new benchmarks: Invalsi MATE to evaluate models performance on mathematical understanding in Italian, Invalsi ITA to evaluate language understanding in Italian and Olimpiadi MATE for more complex mathematical understanding. The first two benchmarks are based on the Invalsi tests, which are administered to students of age between 6 and 18 within the Italian school system and have been validated by several experts in teaching and pedagogy, the third one comes from the Italian high school math Olympics. We evaluate 10 powerful language models on these benchmarks and find that they are bound by 71% accuracy on Invasli MATE, achieved by Llama 3.1 70b instruct and by 88% on Invalsi ITA. For both Invalsi MATE and Invalsi ITA we compare LLMs with the average performance of Italian students to show that Llama 3.1 is the only one to outperform them on Invalsi MATE while most models do so on Invalsi ITA, we then show that Olimpiadi MATE is more challenging than Invalsi MATE and the highest accuracy, achieved by Llama 3.1 405b instruct is 45%. We will make data and evaluation code openly available upon acceptance of the paper.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑