发表机构
Fields Institute; Principles of Intelligence; University of Toronto（菲尔兹研究所; 智能原理机构; 多伦多大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究约1000亿参数的语言模型是否存在可从残差流解码的夜空图表示，发现多数开源模型具备该表示，其性能指标优异且为首个弯曲高维不可约特征流形实例，相关代码已公开。
AI 中文摘要
我们分析了约1000亿参数规模的语言模型是否存在可从其残差流中解码的夜空图表示。研究发现，多数被考察的开源模型确实具备此类表示，且在“夜空中这个天体附近有什么”这类提示词下，该表示常浮现至主成分顶部。除1个模型外，其余模型的该表示在留一法（LOO）测试中均显示出显著得分，可覆盖65%-85%的方差（R²分数），中位角误差低至12°-21°。我们验证了该表示并非来自相关平面表示的简单泄漏。据我们所知，该表示是首个弯曲高维不可约特征流形的实例，论文所用代码已发布在指定网址。
英文摘要
We analyze whether language models of size ~100B have a representation of the night sky map that is decodable from their residual stream. We find that most of the considered open-source models do have such a representation, and it often even surfaces to the top principal components on prompts that ask questions like ``what is close to this object in the night sky''. In all but one model this representation showed significant scores in LOO testing, containing up to 65-85% of variance ($R^2$-score) and having median angular error down to $12^\circ-21^\circ$. We verify that our representation is not a simple leak from a correlated flat representation. To our knowledge, this representation is the first example of a curved high-dimensional irreducible feature manifold. Codes used in the paper are published at https://github.com/l3erdnik/Decodable-sky
