arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.08939cs.AIcs.CL
收藏

前提顺序在大语言模型推理中的重要性

Premise Order Matters in Reasoning with Large Language Models

Google DeepMind(谷歌DeepMind);Stanford University(斯坦福大学)

Xinyun Chen, Ryan A. Chi, Xuezhi Wang, Denny Zhou

AI总结:

本研究揭示大语言模型对前提顺序敏感,顺序与推理步骤一致时性能最佳,打乱顺序可致准确率下降超30%,并发布R-GSM基准验证该效应。

摘要:

大语言模型(LLMs)已在各个领域展现出卓越的推理性能。然而,在推理任务领域,我们发现了一个弱点:尽管前提的顺序并不改变底层任务,但LLMs对前提的顺序却异常脆弱。具体而言,我们观察到,当前提顺序与中间推理步骤所需的上下文一致时,LLMs能达到最佳性能。例如,在演绎推理任务中,以与真实证明相同的顺序呈现前提(而非随机排序)会大幅提高模型的准确率。我们首先在多种LLMs上检验了前提排序对演绎推理的影响,评估结果显示,打乱前提顺序可能导致性能下降超过30%。此外,我们基于GSM8K发布了基准R-GSM,以检验数学问题求解中的排序效应,并再次观察到相对于原始GSM8K基准,准确率显著下降。

英文摘要:

Large language models (LLMs) have accomplished remarkable reasoning performance in various domains. However, in the domain of reasoning tasks, we discover a frailty: LLMs are surprisingly brittle to the ordering of the premises, despite the fact that such ordering does not alter the underlying task. In particular, we observe that LLMs achieve the best performance when the premise order aligns with the context required in intermediate reasoning steps. For example, in deductive reasoning tasks, presenting the premises in the same order as the ground truth proof in the prompt (as opposed to random ordering) drastically increases the model's accuracy. We first examine the effect of premise ordering on deductive reasoning on a variety of LLMs, and our evaluation shows that permuting the premise order can cause a performance drop of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to examine the ordering effect for mathematical problem-solving, and we again observe a significant drop in accuracy, relative to the original GSM8K benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑