arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.17080cs.CL
收藏

MR-GSM8K:用于大型语言模型评估的元推理基准

MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

Chinese University of Hong Kong(香港中文大学);Smartmore Co.Ltd(思谋科技有限公司);Tencent AI Lab(腾讯AI Lab)

Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, Jiaya Jia

AI总结:

本研究提出元推理评估范式,将LLM从问答角色转变为解答评分角色,并基于GSM8K构建MR-GSM8K基准,揭示现有模型在元推理任务上的显著性能差距。

摘要:

在本研究中,我们为大型语言模型(LLMs)引入了一种新颖的评估范式,迫使其从类似学生的传统问答角色转变为类似教师的解答评分角色。该范式聚焦于“对推理的推理”,因此被称为元推理,它将重点从往往忽略推理过程的以结果为导向的评估,转向更全面地评估,从而有效区分不同模型的认知能力。通过在 GSM8K 数据集中应用此范式,我们开发了 MR-GSM8K 基准。我们的广泛分析涵盖了来自开源和商业领域的多个最先进模型,揭示了其训练和评估方法中的根本缺陷。值得注意的是,尽管 Deepseek-v2 和 Claude3-Sonnet 等模型在 GSM8K 中与 GPT-4 竞争激烈,但它们在 MR-GSM8K 中的性能差距却急剧扩大,差异超过 20 个绝对百分点,凸显了我们元推理方法带来的重大挑战。

英文摘要:

In this work, we introduce a novel evaluation paradigm for Large Language Models (LLMs) that compels them to transition from a traditional question-answering role, akin to a student, to a solution-scoring role, akin to a teacher. This paradigm, focusing on "reasoning about reasoning," hence termed meta-reasoning, shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation that effectively distinguishes between the cognitive capabilities of different models. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark. Our extensive analysis includes several state-of-the-art models from both open-source and commercial domains, uncovering fundamental deficiencies in their training and evaluation methodologies. Notably, while models like Deepseek-v2 and Claude3-Sonnet closely competed with GPT-4 in GSM8K, their performance disparities expanded dramatically in MR-GSM8K, with differences widening to over 20 absolute points, underscoring the significant challenge posed by our meta-reasoning approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑