Abstract representational geometry supports inference in large language models
College of Future Information Technology, Fudan University(复旦大学未来信息科技学院);Beijing National Research Center for Information Science and Technology, Tsinghua University(北京信息科学与技术国家研究中心,清华大学)
A defining feature of human intelligence is the ability to adapt to changing environments by inferring latent task structure from sparse observations. Neuroscientific research indicates that this capability relies on the hippocampus constructing abstract representations, expressed as low-dimensional, approximately orthogonal manifolds in neural state space. However, the internal mechanisms of large language models (LLMs) remain largely opaque, making it unclear whether they form comparable abstract representations or instead rely on task-specific statistical regularities when performing comparable reasoning tasks. Here we adapt a contextual reversal-learning paradigm to a text-based setting and compare humans and LLMs at both the Behavioural and representational levels. We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus. Notably, this representational geometry is not uniformly distributed but is organized hierarchically across model depth: whereas lower layers show early, stable encoding of stimulus identity, higher layers form a hippocampal-like functional band enriched for abstract context geometry associated with inference. Furthermore, complementary intervention experiments mechanistically implicate geometry in reasoning: task-sequence language modelling induces geometric disentanglement, whereas geometric regularization of higher layers increases the emergence of generalizable inference. Together, these findings establish abstract representational geometry as a mechanistic principle supporting inference in large language models.
Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models
Human Centric ART, University of Rome Tor Vergata(人类中心ART,罗马大学托尔维加塔分校);ILCC, School of Informatics, University of Edinburgh(ILCC,信息学院,爱丁堡大学);School of Computer Science, University of Sheffield(计算机科学学院,谢菲尔德大学);Almawave S.p.A.(Almawave公司)
Beyond representational alignment with brain-guided language models for robust reasoning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(北京大学通用人工智能国家重点实验室、智能科学与技术学院);Department of Psychological and Cognitive Sciences, Tsinghua University(清华大学心理与认知科学系);Microsoft Research Asia(微软亚洲研究院)
Blind to Position, Biased in Language: Probing Mid-Layer Representational Bias in Vision-Language Encoders for Zero-Shot Language-Grounded Spatial Understanding
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);Samsung Electronics(三星电子)
BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction
University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Convergent Representations of Linguistic Constructions in Human and Artificial Neural Systems
Department of English and American Studies, University Erlangen-Nuremberg(英语与美国研究系,埃尔朗根-纽伦堡大学);Pattern Recognition Lab, University Erlangen-Nuremberg(模式识别实验室,埃尔朗根-纽伦堡大学);Neuromodulation and Neuroprosthetics, University Hospital Mannheim, University Heidelberg(神经调控与神经假体,曼海姆大学医院,海德堡大学);BGU Ludwigshafen, Germany(吕贝克大学吕贝克分校,德国);Neuroscience Lab, University Hospital Erlangen(神经科学实验室,埃尔朗根大学医院)