arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.10786cs.AIcs.CL
收藏

一次评估多个问题的LLM

Evaluating LLMs with Multiple Problems at once

Department of Linguistics & Institute for Advanced Computational Science(语言学系及高级计算科学研究所)

Zhengxiang Wang, Jordan Kodner, Owen Rambow

AI总结:

本文提出多问题评估(MPE)范式,构建ZeMPE基准(含53,100个零样本提示),在13个LLM上系统评估,发现LLM能处理多问题但存在条件限制,并发布语料与代码。

摘要:

本文展示了同时用多个问题评估LLM(大语言模型)的益处与成效,我们将这一范式称为多问题评估(MPE)。与传统的单问题评估(提示中呈现单个问题并期望一个特定答案)不同,MPE将多个问题置于同一提示中,并评估LLM在单次输出中回答所有这些问题的能力。利用已有的6个分类基准和12个推理基准,我们引入了一个名为ZeMPE(零样本多问题评估)的新基准,包含53,100个零样本多问题提示。我们在ZeMPE上对来自5个模型家族的共13个LLM进行了实验,以呈现全面且系统的MPE研究。我们的结果表明,LLM能够处理来自单一数据源的多个问题,也能分别处理这些问题,但在某些条件下,这种多问题处理能力会有所不足。此外,我们进行了深入的进一步分析,并探索了可能赋予LLM多问题处理能力的模型层面因素。我们发布了我们的语料库和代码,以促进未来的研究。

英文摘要:

This paper shows the benefits and fruitfulness of evaluating LLMs with multiple problems at once, a paradigm we call multi-problem evaluation (MPE). Unlike conventional single-problem evaluation, where a prompt presents a single problem and expects one specific answer, MPE places multiple problems together in a single prompt and assesses how well an LLM answers all these problems in a single output. Leveraging 6 classification and 12 reasoning benchmarks that already exist, we introduce a new benchmark called ZeMPE (Zero-shot Multi-Problem Evaluation), comprising 53,100 zero-shot multi-problem prompts. We experiment with a total of 13 LLMs from 5 model families on ZeMPE to present a comprehensive and systematic MPE. Our results show that LLMs are capable of handling multiple problems from a single data source as well as handling them separately, but there are conditions this multiple problem handling capability falls short. In addition, we perform in-depth further analyses and explore model-level factors that may enable multiple problem handling capabilities in LLMs. We release our corpus and code to facilitate future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑