你的大型语言模型是知识渊博，还是仅靠选项作弊？
Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?
AI总结：
本文通过图挖掘从 UnifiedQA 构建820题对比集，测试12个LLM后发现其高MCQA排名并非主要依赖仅看选项的捷径。
摘要：
近期研究表明，大型语言模型（LLM）可以仅凭选项回答多项选择题，但这是否意味着 LLM 在 MCQA 排行榜上的排名在很大程度上受到其在仅含选项设置下能力的影响？为回答这一问题，我们使用一个对比集来探测 LLM 在 MCQA 中是否过度依赖仅含选项的捷径。以往工作通过昂贵的人工标注或可能存在偏差的模型生成数据构建对比集，而我们采用图挖掘从现有 MCQA 数据集中提取对比集。我们将该方法用于 UnifiedQA——一组包含六个常识推理数据集、且仅含选项准确率较高的集合——构建了一个含 820 个问题的对比集。在验证该对比集后，我们测试了 12 个 LLM，发现这些模型在同时获得问题和选项时，并未表现出对仅含选项捷径的依赖。因此，尽管 MCQA 容易出现较高的仅含选项准确率，我们认为 LLM 在 MCQA 排行榜上获得高排名并不仅仅是因为它们能够利用仅含选项的捷径。
英文摘要：
Recent work shows that large language models (LLMs) can answer multiple-choice questions using only the choices, but does this mean that MCQA leaderboard rankings of LLMs are largely influenced by abilities in choices-only settings? To answer this, we use a contrast set that probes if LLMs over-rely on choices-only shortcuts in MCQA. While previous works build contrast sets via expensive human annotations or model-generated data which can be biased, we employ graph mining to extract contrast sets from existing MCQA datasets. We use our method on UnifiedQA, a group of six commonsense reasoning datasets with high choices-only accuracy, to build an 820-question contrast set. After validating our contrast set, we test 12 LLMs, finding that these models do not exhibit reliance on choice-only shortcuts when given both the question and choices. Thus, despite the susceptibility~of MCQA to high choices-only accuracy, we argue that LLMs are not obtaining high ranks on MCQA leaderboards just due to their ability to exploit choices-only shortcuts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
