arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.03439cs.CLcs.AI
收藏

评估ChatGPT与GPT-4的逻辑推理能力

Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

Westlake University（西湖大学）；Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）；Fudan University（复旦大学）

Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, Yue Zhang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究通过多个逻辑推理基准及自建分布外数据集测试ChatGPT和GPT-4的推理能力，发现二者在知名数据集表现较好但新数据集和分布外场景性能下降，同时发布了LogiEval基准套件。

摘要：

掌握逻辑推理能力是一项综合性的自然语言理解工作。随着被强调在推理任务中具备“先进”能力的Generative Pretrained Transformer 4（GPT-4，生成式预训练Transformer 4）的发布，我们迫切想要了解GPT-4在各类逻辑推理任务上的表现。本报告分析了多个逻辑推理数据集，包括LogiQA、ReClor等流行基准，以及AR-LSAT等新发布数据集。我们用需要逻辑推理的基准测试了多项选择阅读理解和自然语言推理任务。我们进一步构建了一个逻辑推理分布外数据集，以探究ChatGPT和GPT-4的鲁棒性，同时还对二者的性能进行了对比。实验结果表明，在大多数逻辑推理基准上，ChatGPT的表现显著优于RoBERTa微调方法。通过提前获取GPT-4 API的访问权限，我们得以对GPT-4模型开展深入实验，结果显示GPT-4在大多数逻辑推理数据集上取得了更优的性能。在各基准中，ChatGPT和GPT-4在LogiQA、ReClor这类知名数据集上表现相对较好，但在处理新发布数据集和分布外数据集时性能显著下降。逻辑推理对ChatGPT和GPT-4而言仍具挑战性，尤其是在分布外数据集和自然语言推理数据集上。我们发布了提示式逻辑推理数据集作为基准套件，并将其命名为LogiEval。

英文摘要：

Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑