In the field of NLP, Large Language Models (LLMs) have markedly enhanced performance across a variety of tasks. However, the comprehensive evaluation of LLMs remains an inevitable challenge for the community. Recently, the adoption of Multiple Choice Question Answering (MCQA) as a benchmark for assessing LLMs has gained considerable traction. However, concerns regarding the robustness of this evaluative method persist. Building upon previous discussions on the issue of \textit{variability}, we reveal an additional dimension of concern: LLMs may perform MCQA by selecting the least incorrect option rather than distinctly correct. This observation suggests that LLMs might regard multiple options as correct, which could undermine the reliability of MCQA as a metric for evaluating LLMs. To address this challenge, we introduce an enhanced dataset augmentation method for MCQA, termed MCQA+, to provide a more accurate reflection of the model performance, thereby highlighting the necessity for more sophisticated evaluation mechanisms in the assessment of LLM capabilities.
Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Options
University of Warsaw(华沙大学);Institute of Mathematics, Polish Academy of Sciences(波兰科学院数学研究所);MIM Solutions(MIM解决方案);K-Scale Labs(K-Scale实验室);IDEAS NCBR;IDEAS Research Institute(IDEAS研究学院)
ELLA: Empowering LLMs for Interpretable, Accurate and Informative Legal Advice
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院)
Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning
Department of Computer Science(计算机科学系);Institute For Artificial Intelligence(人工智能研究所);Hamm-Lippstadt University of Applied Sciences(哈姆-利普施塔特应用科学大学);University of Stuttgart(斯图加特大学)
Reasoning on Efficient Knowledge Paths:Knowledge Graph Guides Large Language Model for Domain Question Answering
State Key Laboratory of Media Convergence Production Technology and Systems(媒体融合生产技术与系统国家重点实验室);Institute of Computer Science and Technology, Peking University(北京大学计算机学院)
Corpus Prevalence of Multiple-Choice Question Options
Center for Language and Cognition, University of Groningen(语言与认知中心,格罗宁根大学);Department of Experimental Psychology, University of Groningen(实验心理学系,格罗宁根大学)