arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.02678cs.AIcs.CL
收藏

大型语言模型中的推理:一种几何视角

Reasoning in Large Language Models: A Geometric Perspective

Tenyx(滕尼克斯(Tenyx))

Romain Cosentino, Sarath Shekkizhar

AI总结:

本文从几何视角研究LLM推理能力,提出自注意力图密度决定MLP输入内在维度,并证明更高内在维度带来更强表达能力,为推理增强方法提供理论解释。

摘要:

大型语言模型(LLMs)在现实世界应用中的进步关键取决于其推理能力的增强。在本工作中,我们通过LLMs的几何理解来探索其推理能力。我们建立了LLMs的表达能力与其自注意力图密度之间的联系。我们的分析表明,这些图的密度定义了MLP块输入的内在维度。我们通过理论分析和玩具示例证明,更高的内在维度意味着LLM具有更强的表达能力。我们进一步提供了经验证据,将这一几何框架与近期旨在增强LLMs推理能力的方法进展联系起来。

英文摘要:

The advancement of large language models (LLMs) for real-world applications hinges critically on enhancing their reasoning capabilities. In this work, we explore the reasoning abilities of large language models (LLMs) through their geometrical understanding. We establish a connection between the expressive power of LLMs and the density of their self-attention graphs. Our analysis demonstrates that the density of these graphs defines the intrinsic dimension of the inputs to the MLP blocks. We demonstrate through theoretical analysis and toy examples that a higher intrinsic dimension implies a greater expressive capacity of the LLM. We further provide empirical evidence linking this geometric framework to recent advancements in methods aimed at enhancing the reasoning capabilities of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑