arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01349cs.CLcs.AI
收藏

LLMs 可能通过选择错误最少的选项来执行 MCQA

LLMs May Perform MCQA by Selecting the Least Incorrect Option

Harbin Institute of Technology(哈尔滨工业大学);Research Center for Social Computing and Information Retrieval(社会计算与信息检索研究中心)

Haochun Wang, Sendong Zhao, Zewen Qiang, Nuwa Xi, Bing Qin, Ting Liu

AI总结:

针对 MCQA 可能无法区分“正确”与“错误最少”选项的问题,本文提出 MCQA+ 数据集增强方法,以更可靠地评估 LLM 能力。

摘要:

在 NLP 领域,Large Language Models(LLMs,大语言模型)显著提升了多种任务的性能。然而,对 LLMs 的全面评估仍是该领域不可避免的挑战。近来,采用 Multiple Choice Question Answering(MCQA,多项选择题问答)作为评估 LLMs 的基准已获得相当大的关注。但是,关于这种评估方法鲁棒性的担忧仍然存在。基于此前关于“可变性”问题的讨论,我们揭示了另一个值得关注的维度:LLMs 可能通过选择错误最少的选项,而非明确正确的选项来执行 MCQA。这一观察表明,LLMs 可能将多个选项视为正确,从而削弱 MCQA 作为评估 LLMs 指标的可靠性。为应对这一挑战,我们提出一种增强的 MCQA 数据集增强方法,称为 MCQA+,以更准确地反映模型性能,从而凸显在评估 LLM 能力时需要更复杂评估机制的必要性。

英文摘要:

In the field of NLP, Large Language Models (LLMs) have markedly enhanced performance across a variety of tasks. However, the comprehensive evaluation of LLMs remains an inevitable challenge for the community. Recently, the adoption of Multiple Choice Question Answering (MCQA) as a benchmark for assessing LLMs has gained considerable traction. However, concerns regarding the robustness of this evaluative method persist. Building upon previous discussions on the issue of \textit{variability}, we reveal an additional dimension of concern: LLMs may perform MCQA by selecting the least incorrect option rather than distinctly correct. This observation suggests that LLMs might regard multiple options as correct, which could undermine the reliability of MCQA as a metric for evaluating LLMs. To address this challenge, we introduce an enhanced dataset augmentation method for MCQA, termed MCQA+, to provide a more accurate reflection of the model performance, thereby highlighting the necessity for more sophisticated evaluation mechanisms in the assessment of LLM capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑