arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.23345cs.AI
收藏

抽象表征几何支持大型语言模型中的推理

Abstract representational geometry supports inference in large language models

College of Future Information Technology, Fudan University(复旦大学未来信息科技学院);Beijing National Research Center for Information Science and Technology, Tsinghua University(北京信息科学与技术国家研究中心,清华大学)

Yunan Zeng, Yuwang Wang

AI总结:

本研究通过文本版情境反转学习范式,比较人类与LLM的行为和表征,发现LLM内部状态在推理时形成类似海马体的抽象几何结构,且该结构分层组织,高层次表征与推理相关。

摘要:

人类智能的一个定义性特征是通过从稀疏观察中推断潜在任务结构来适应变化环境的能力。神经科学研究表明,这种能力依赖于海马体构建抽象表征,表现为神经状态空间中低维、近似正交的流形。然而,大型语言模型(LLM)的内部机制在很大程度上仍不透明,使得不清楚它们在执行类似推理任务时是形成可比较的抽象表征,还是依赖于任务特定的统计规律。在这里,我们将情境反转学习范式改编为基于文本的设置,并在行为和表征层面比较人类和LLM。我们报告说,尽管LLM表现出可泛化推理的频率低于人类,但当这种推理发生时,它们的内部状态表现出与海马体中报道的相似的抽象几何结构。值得注意的是,这种表征几何并非均匀分布,而是按模型深度分层组织:较低层显示早期、稳定的刺激身份编码,而较高层形成一个类似海马体的功能带,富含与推理相关的抽象情境几何。此外,互补的干预实验从机制上将几何与推理联系起来:任务序列语言建模诱导几何解缠,而较高层的几何正则化增加了可泛化推理的出现。总之,这些发现确立了抽象表征几何作为支持大型语言模型中推理的机制原理。

英文摘要:

A defining feature of human intelligence is the ability to adapt to changing environments by inferring latent task structure from sparse observations. Neuroscientific research indicates that this capability relies on the hippocampus constructing abstract representations, expressed as low-dimensional, approximately orthogonal manifolds in neural state space. However, the internal mechanisms of large language models (LLMs) remain largely opaque, making it unclear whether they form comparable abstract representations or instead rely on task-specific statistical regularities when performing comparable reasoning tasks. Here we adapt a contextual reversal-learning paradigm to a text-based setting and compare humans and LLMs at both the Behavioural and representational levels. We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus. Notably, this representational geometry is not uniformly distributed but is organized hierarchically across model depth: whereas lower layers show early, stable encoding of stimulus identity, higher layers form a hippocampal-like functional band enriched for abstract context geometry associated with inference. Furthermore, complementary intervention experiments mechanistically implicate geometry in reasoning: task-sequence language modelling induces geometric disentanglement, whereas geometric regularization of higher layers increases the emergence of generalizable inference. Together, these findings establish abstract representational geometry as a mechanistic principle supporting inference in large language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑