arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.16689cs.LG
收藏

预测统计塑造网格行走者的涌现世界表征

Predictive Statistics Shape Emergent World Representations of Grid Walkers

Max Planck Institute for Human Cognitive and Brain Sciences(马克斯·普朗克人类认知与脑科学研究所);Leipzig University(莱比锡大学);ScaDS.AI

Sasha Brenner, Thomas R. Knösche, Nico Scherf

AI总结:

通过约束随机游走实验,发现解码器仅Transformer的第一注意力块提取预测充分统计量,后续层将其转化为预测几何,形成可读的世界模型,而循环网络未分离此阶段。

摘要:

下一个词预测器通常似乎会发展出对潜在世界及其规则的内部表征。这些模型的概率性质表明世界结构与概率分布几何之间存在深层联系。为了更精确地理解这种联系,我们使用一个最小随机过程作为受控设置:在二维格点上的约束随机游走,必须在预定步数后到达固定终点。对该过程的最优预测仅取决于由游走者相对于目标的位置和剩余时间范围决定的充分向量;换句话说,概率分布由世界的网格几何参数化。我们在从这些游走的精确分布中采样的前缀上训练解码器仅Transformer和循环网络,并通过跨层测量对齐和线性可读性,将其隐藏激活与预测的充分统计量进行比较。我们发现Transformer的计算分为两个阶段:第一个注意力块从输入中提取充分统计量,后续层将其转化为下一步预测几何。在不同约束变体中,注意力后的表征是通用的:一个共享的晶格世界状态,可以直接作为世界模型读取,追溯到数据的预测几何。后续层随后将其专门化到每个变体的下一步分布。循环网络达到相同的贝叶斯最优损失,但未将这个世界状态隔离为一个单独阶段,表明世界模型几何也依赖于架构。尽管在玩具系统中演示,结果表明预测分布的几何是理解神经网络如何内化其数据结构的有用视角。

英文摘要:

Next-token predictors often appear to develop internal representations of the latent world and its rules. The probabilistic nature of these models suggests a deep connection between the structure of the world and the geometry of probability distributions. In order to understand this link more precisely, we use a minimal stochastic process as a controlled setting: constrained random walks on a two-dimensional lattice that must reach a fixed endpoint after a predetermined number of steps. Optimal prediction of this process solely depends on a sufficient vector determined by the walker's position relative to the target and the remaining time horizon; in other words, the probability distributions are parametrized by the world's grid geometry. We train decoder-only transformers and recurrent networks on prefixes sampled from the exact distribution of these walks and compare their hidden activations to sufficient statistics of prediction, by measuring alignment and linear readability across layers. We find that the transformer's computation factors into two stages: the first attention block extracts the sufficient statistic from the input, and later layers transform it into the next-step predictive geometry. Across constraint variants the post-attention representation is universal: a shared world-state of the lattice that can be read directly as a world model, traced to the predictive geometry of the data. Later layers then specialize it to each variant's next-step distribution. Recurrent networks reach the same Bayes-optimal loss but do not isolate this world-state as a separate stage, showing that the world-model geometry also depends on architecture. Although demonstrated in a toy system, the results suggest that the geometry of the predictive distribution is a useful lens on how neural networks internalize the structure of their data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑