arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.11467cs.CL
收藏

大型语言模型的过度推理与冗余计算

Over-Reasoning and Redundant Calculation of Large Language Models

National Taiwan University(台湾大学)

Cheng-Han Chiang, Hung-yi Lee

AI总结:

本文发现LLMs在无需计算的GSM8K-Zero数据集上存在过度推理与冗余计算现象,并通过实验探究其成因。

摘要:

大型语言模型(LLMs)能够逐步解决问题。尽管这种思维链(CoT)推理提升了LLMs的性能,但LLMs是否知道何时使用CoT以及这些CoT对于回答问题是否总是必要的,仍不清楚。本文表明,LLMs在人工构建的数学问答数据集GSM8K-Zero上倾向于生成冗余计算和推理。GSM8K-Zero的构建使得问题无需任何计算即可回答,但包括Llama-2模型和Claude-2在内的LLMs,倾向于生成冗长且不必要的计算来回答问题。我们还进行了实验以解释为什么LLMs会生成冗余计算和推理。GSM8K-Zero已公开发布于https://github.com/d223302/Over-Reasoning-of-LLMs和https://huggingface.co/datasets/dcml0714/GSM8K-Zero。

英文摘要:

Large language models (LLMs) can solve problems step-by-step. While this chain-of-thought (CoT) reasoning boosts LLMs' performance, it is unclear if LLMs \textit{know} when to use CoT and whether those CoT are always necessary to answer the question. This paper shows that LLMs tend to generate redundant calculations and reasoning on a manually constructed math QA dataset, GSM8K-Zero. GSM8K-Zero is constructed such that the questions can be answered without any calculations, but LLMs, including Llama-2 models and Claude-2, tend to generate lengthy and unnecessary calculations to answer the questions. We also conduct experiments to explain why LLMs generate redundant calculations and reasonings. GSM8K-Zero is publicly available at https://github.com/d223302/Over-Reasoning-of-LLMs and https://huggingface.co/datasets/dcml0714/GSM8K-Zero.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑