AraSTEM：一个用于评估大语言模型在STEM学科中知识的原生阿拉伯语多项选择基准
AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects
AI总结：
本文提出AraSTEM阿拉伯语STEM多项选择基准，用于评估大语言模型在非英语科学知识上的表现，实验表明现有公开模型普遍表现不佳，凸显本地化语言模型的必要性。
摘要：
大语言模型（LLMs）展现出了卓越的能力，不仅能够生成类似人类的文本，还能获取知识。这凸显了超越典型自然语言处理下游基准、评估LLMs包括知识和推理在内的各个方面的必要性。目前已开发出大量基准用于评估LLMs的知识，但它们主要聚焦于英语。鉴于许多LLMs是多语言的，仅依赖英语知识基准评估是不够的。为解决这一问题，我们提出了AraSTEM，这是一个新的阿拉伯语多项选择数据集，旨在评估LLMs在STEM学科中的知识。该数据集涵盖不同层次的一系列主题，要求模型展现出对科学阿拉伯语的深入理解才能获得高准确率。我们的研究结果表明，不同规模的公开可用模型在该数据集上均表现困难，这强调了需要更多本地化语言模型。该数据集可在Hugging Face上免费获取。
英文摘要：
Large Language Models (LLMs) have shown remarkable capabilities, not only in generating human-like text, but also in acquiring knowledge. This highlights the need to go beyond the typical Natural Language Processing downstream benchmarks and asses the various aspects of LLMs including knowledge and reasoning. Numerous benchmarks have been developed to evaluate LLMs knowledge, but they predominantly focus on the English language. Given that many LLMs are multilingual, relying solely on benchmarking English knowledge is insufficient. To address this issue, we introduce AraSTEM, a new Arabic multiple-choice question dataset aimed at evaluating LLMs knowledge in STEM subjects. The dataset spans a range of topics at different levels which requires models to demonstrate a deep understanding of scientific Arabic in order to achieve high accuracy. Our findings show that publicly available models of varying sizes struggle with this dataset, and underscores the need for more localized language models. The dataset is freely accessible on Hugging Face.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
