从柏拉图到亚里士多德:迈向表征假说的通用理论
From Plato to Aristotle: Toward a General Theory of Representation Hypotheses
AI总结:
该研究提出格表征假说(LRH)统一不同表征对齐观点,分析柏拉图、亚里士多德等表征假说的差异,经多模型实验验证,为表征假说通用理论奠定基础。
摘要:
柏拉图式表征假说(PRH)认为,能力不断提升的模型会收敛到共同表征。然而,近期研究对通过邻域、生成树和距离对齐表征提出了不同观点。我们表明,这些观点存在差异是因为其背后的对齐标准处于同一个格的不同位置,我们将这一解释形式化为格表征假说(LRH)。在LRH框架下,PRH所主张的收敛是在格顶部达成一致,而其证据来自邻域重叠,这是一种接近格底部的标准。亚里士多德式表征假说(ARH)将收敛限制在局部邻域,因为校准使得邻域重叠随模型规模的增长保持稳定,但在很大程度上消除了中心核对齐的增长。我们表明,这种消除可来自对层最大值的校准,因此全局收敛仍有待研究。回到柏拉图洞穴(BPC)从更大评估集上邻域重叠的崩溃中推断出粗略的共享结构,而我们将其至少部分归因于固定数量的邻域覆盖了更短的距离。我们表明,几何结构之上的结构(SOG)所使用的生成树重叠大多重复了一近邻重叠。LRH也可扩展到独立训练的模型,其训练过程和目标决定了跨运行可靠满足的标准。我们针对线性自编码器的训练过程和有限对比吉布斯模型的目标证明了这一主张。对数十个预训练视觉和语言模型以及数千次理论模型运行的实验支持了这些结论。我们希望这项工作为表征假说的通用理论铺平道路,提供一个分析它们并比较其证据的框架。项目页面:this https URL
英文摘要:
The Platonic Representation Hypothesis (PRH) holds that increasingly capable models converge toward a common representation. However, recent studies have shown different views on aligning representations via their neighborhoods, spanning trees, and distances. We show that these views can differ because the alignment criteria behind them sit at different points of one lattice, and we formalize this explanation as the Lattice Representation Hypothesis (LRH). Under the LRH, the convergence claimed by the PRH is agreement at the top of the lattice, whereas its evidence comes from neighbor overlap, a criterion near the bottom. The Aristotelian Representation Hypothesis (ARH) restricts convergence to local neighborhoods because calibration keeps the growth of neighbor overlap with model size but largely removes that of centered kernel alignment. We show that this removal can come from calibrating a maximum over layers, so global convergence remains open. Back into Plato's Cave (BPC) infers coarse shared structure from the collapse of neighbor overlap on larger evaluation sets, whereas we trace it at least partly to a fixed number of neighbors covering shorter distances. We show that the spanning-tree overlap used by Structure over Geometry (SOG) mostly repeats one-nearest-neighbor overlap. The LRH also extends to independently trained models, whose training procedure and objective decide the criteria met reliably across runs. We prove this claim for the training procedure in a linear autoencoder and for the objective in a finite contrastive Gibbs model. Experiments on dozens of pretrained vision and language models and thousands of runs of both theoretical models support these conclusions. We hope this work paves the way toward a general theory of representation hypotheses, with a framework to analyze them and compare their evidence. Project page: https://eethanshi.github.io/LRH
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。