arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.01190cs.LGstat.ML
收藏

通过记忆能力分析思维链的威力

Analyzing the Power of Chain of Thought through Memorization Capabilities

Institute of AI for Industries(人工智能产业研究院);State Key Laboratory of Mathematical Sciences(数学科学国家重点实验室);Academy of Mathematics and Systems Science(数学与系统科学学院);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of System Software of Chinese Academy of Sciences(中国科学院系统软件重点实验室)

Lijia Yu, Xiao-Shan Gao, Lijun Zhang

AI总结:

本文通过记忆能力视角分析思维链(CoT),证明带有或不带CoT的固定精度transformers记忆有限推理数据集的参数量上下界均为$overlineΘ(N)$,得出CoT并不能在所有推理任务上增强transformers推理能力的结论。

摘要:

已有研究表明,思维链(CoT)能够增强大型语言模型(LLMs)解决某些数学推理问题的能力。然而,CoT的能力仍未被完全探索。作为一个重要的例子,以下基本问题尚未得到解答:CoT是否在所有推理任务上都扩展了transformers的能力?我们证明,使用transformers进行推理本质上是一个针对推理数据集的记忆问题。因此,考察CoT在所有推理任务上的威力就等同于分析CoT transformers的记忆能力。在本文中,我们对带有或不带有CoT的固定精度transformers的记忆能力给出了完整的描述,并对上述问题给出了否定的回答。具体而言,我们首先给出了带有和不带有CoT的固定精度transformers记忆有限推理数据集的充分必要条件,并表明这两个条件互不蕴含。然后,我们给出了带有或不带有CoT的transformers记忆包含$N$个元素的有限推理数据集所需参数数量的下界和上界,在所有情况下均为$overlineΘ(N)$。这意味着存在某些推理任务,CoT并不能增强transformers的推理能力,从而对上述问题给出了否定的回答。最后,我们给出了关于CoT transformers记忆无限推理数据集的首批结果,并表明一些简单的无限数据集无法被带有或不带有CoT的transformers所记忆。

英文摘要:

It has been shown that the chain of thought (CoT) can enhance the power of large language models (LLMs) to solve certain mathematical reasoning problems. However, the capacity of CoT is still not fully explored. As an important instance, the following basic question has not yet been answered: Does CoT expand the capability of transformers across all reasoning tasks? We demonstrate that reasoning with transformers is essentially a memorization problem for reasoning datasets. Thus, examining the power of CoT across all reasoning tasks amounts to analyzing the memorization capabilities of CoT transformers. In this paper, we give a complete description of the memorization capabilities of fixed-precision transformers with or without CoT and give a negative answer to the above-mentioned question. Precisely, we first give necessary and sufficient conditions for fixed-precision transformers with and without CoT to memorize a finite reasoning dataset and show that these two conditions do not imply each other. Then, we give lower and upper bounds for the number of parameters needed for transformers with or without CoT to memorize a finite reasoning dataset with $N$ elements, which are $\overlineΘ(N)$ in all cases. This implies that there exist reasoning tasks for which CoT does not enhance the reasoning power of transformers, leading to a negative answer to the above-mentioned question. Finally, we give the first results on memorizing infinite reasoning datasets by CoT transformers and show that some simple infinite datasets cannot be memorized by transformers with or without CoT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑