GAOKAO-Eval：高分真的能反映大语言模型的强能力吗？
GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?
AI总结：
该研究构建了基于中国高考的GAOKAO-Eval基准开展闭卷评估，发现LLM高分仍无法真实反映类人能力，借助Rasch模型分析出得分模式差异，并指出o1的推理机制可缓解该不匹配。
摘要：
大语言模型（LLM）通常采用人工构建的基准进行评估，其前提是更高的分数隐含着更强的类人表现。然而，越来越多的人担忧，LLM可能会因数据泄露而“钻基准的空子”，取得高分却在对人类而言很简单的任务上表现不佳。为切实解决这一问题，我们构建了基于中国全国高等学校招生统一考试（高考）的综合性基准GAOKAO-Eval，并对高考前发布的代表性模型开展“闭卷”评估。与普遍共识相反，即便在排除数据泄露问题、保障评估全面性后，GAOKAO-Eval仍显示高分无法真正反映与人类对齐的能力。为更好地理解这种不匹配，我们引入认知心理学中的Rasch模型（拉施模型）分析LLM的得分模式，识别出两大关键差异：1）在不同难度题目上表现异常稳定；2）在难度相近的题目上表现方差极大。此外，我们还发现教师对LLM生成答案的评分存在不一致性，且LLM存在反复出现的错误模式。我们发现这些现象与OpenAI o1背后的设计动机高度契合，而o1的“推理即难度”机制能够缓解这种不匹配。这些结果表明，GAOKAO-Eval能够揭示现有基准未捕捉到的LLM能力局限，凸显了开展更适配LLM的难度分析的必要性。
英文摘要：
Large Language Models (LLMs) are commonly evaluated using human-crafted benchmarks, under the premise that higher scores implicitly reflect stronger human-like performance. However, there is growing concern that LLMs may ``game" these benchmarks due to data leakage, achieving high scores while struggling with tasks simple for humans. To substantively address the problem, we create GAOKAO-Eval, a comprehensive benchmark based on China's National College Entrance Examination (Gaokao), and conduct ``closed-book" evaluations for representative models released prior to Gaokao. Contrary to prevailing consensus, even after addressing data leakage and comprehensiveness, GAOKAO-Eval reveals that high scores still fail to truly reflect human-aligned capabilities. To better understand this mismatch, We introduce the Rasch model from cognitive psychology to analyze LLM scoring patterns and identify two key discrepancies: 1) anomalous consistent performance across various question difficulties, and 2) high variance in performance on questions of similar difficulty. In addition, We identified inconsistent grading of LLM-generated answers among teachers and recurring mistake patterns. we find that the phenomenons are well-grounded in the motivations behind OpenAI o1, and o1's reasoning-as-difficulties can mitigate the mismatch. These results show that GAOKAO-Eval can reveal limitations in LLM capabilities not captured by current benchmarks and highlight the need for more LLM-aligned difficulty analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
