AI 中文总结
本文质疑相似度分数作为结构证据的有效性，通过探针跨语言迁移和大脑对齐实验，揭示分数可能源于工具失效或非特异因素，强调需评估无对应关系时的基线分数。
AI 中文摘要
研究人员常常通过报告相似度分数来支持模型与大脑或跨语言共享结构的观点。我们探究当共享结构不存在，或测量工具失效时，该分数意味着什么。我们检验了两种情境，在两种情境中分数均非表面所示。首先，一个用于区分一种语言中语法与不合语法句子的探针，在迁移至更远的语言时表现更差，这通常被视为共享结构的证据。但探针本身沿同一轴线变差：在十七种语言中，四种语言的表现仅达随机水平，因此二百七十二个语言对中有六十四个是用失效工具评分的。剔除这些语言会使关系强度减半，但它们也是最远的语言，此设计无法区分两种效应。计数方式同样重要：同一数据在将二百七十二对视为独立时给出p=0.0006，而在以十七种语言为正确单位时p=0.155。其次，训练语言模型以匹配人类大脑反应，将其相似度分数从0.10提升至0.34，而天花板为0.54。一个在目标与大脑对应关系被破坏的数据上训练的模型仍得0.31分，因此上升中仅0.028至0.068是大脑特异的。即使没有模型，被破坏的目标与真实目标已有0.204的基线，该基线随目标排名除以句子数量而变化。最后，沿语言自身方向引导语言有效（在十七种语言中的十六种中，相对于随机方向提升6.2），但未显示随语言距离变化的效应。在询问对应关系是否有帮助之前，应先问没有它时分数能保留多少。
英文摘要
Similarity scores are often offered as evidence that a model shares structure with the brain or across languages. We ask what such a score reads when that structure is removed, or when the instrument measuring it does not work, in two settings. Across seventeen languages, a grammaticality probe transfers worse between more distant languages (r = -0.66). But the probe's own accuracy falls along the same axis and is at chance in four languages, four of the five most distant. Dropping them halves the explained variance, and because it also narrows the range of distances, the design cannot say how much of the gradient is the instrument. Counting the 272 language pairs as independent gives p = 0.0006 for a steering effect that is null when the seventeen languages are the unit (p = 0.155). In brain alignment, a training objective raises a language model's similarity to fMRI responses from 0.10 to 0.34 (reliability ceiling 0.54), yet targets with the brain correspondence destroyed still reach 0.31. A shuffled target scores about k/n against the real one, for rank k and n sentences, a baseline that can be computed before any model is trained. Both interventions work, yet we detect no distance-graded steering effect and no syntactic benefit from alignment. Before reading a correspondence score, measure how much of it survives without the correspondence.
Comments12 pages, 2 figures, 1 table. Accepted as a poster at the NeurIPS 2026 - Linguistic Principles for Foundation Models (LP4FM). Code: https://github.com/saman-rahbar/score-is-not-the-structure
