arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.29634cs.LG
收藏

Transformer中的关系秩几何:检测与引导隐藏状态关系框架

Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Frames

Independent Researcher(独立研究者)

Mazen Kobrosly

AI总结:

本文通过Plücker符号熵检测Transformer隐藏状态中元组关系的秩索引几何,并在Llama系列模型上验证了关系探测的可控干预,实现了从关系探测到关系框架干预的受控桥梁。

摘要:

Transformer隐藏状态通常通过局部或低阶对象解释:神经元、稀疏特征、注意力头、残差流方向或激活补丁。本文研究一个互补对象:元组间关系的秩索引几何。我使用Plücker符号熵来测试r元关系是否在隐藏状态空间中留下arity匹配的方向签名。在Llama系列8B、70B和405B检查点上,真实关系元组在预期秩k=r(r=3,...,6)处显示出比随机控制审计中打乱元组更强的方向签名一致性。多模板审计表明,这些效应在表面变化下仍然存在,所有测试的405B行保持正预期秩边际,8B/70B保持正行,但带有构造器特定的混合单元。然后我问相同的关系几何是否可以被引导。在一个边缘网格干净/损坏干预实验中,使用32个提示,行/列框架和答案格式保持不变,而YES/NO关系图发生变化,损坏的隐藏状态关系框架被修补为干净或安慰剂目标。在70B和405B中,干净目标的关系框架路径恢复了干净答案行为和残差关系几何,而仅质心和等范数控制显示出可忽略的恢复。位置/顺序控制进一步将标记点重要性从有序干净框架几何中分离:目标干净形状和跨提示干净形状在标记接口处恢复行为和残差几何,而损坏供体转移、同位置置换/反射、错误位置干净增量、仅质心运动和等范数噪声失败或远低于干净框架路径。结果是从关系探测到关系框架干预的受控桥梁:关系秩几何可以在Transformer隐藏状态中被检测、定位和行为验证。

英文摘要:

Transformer hidden states are often interpreted through local or low-order objects: neurons, sparse features, attention heads, residual-stream directions, or activation patches. This paper studies a complementary object: the rank-indexed geometry of relations among token tuples. I use Plucker sign entropy to test whether r-argument relations leave arity-matched orientation signatures in hidden-state space. Across Llama-family 8B, 70B, and 405B checkpoints, true relation tuples show stronger orientation-sign consistency at the expected rank k=r for r=3,...,6 than scrambled tuples under matched random-control audits. Multi-template audits show that the effects survive surface variation, with all tested 405B rows retaining positive expected-rank margins and 8B/70B retaining positive rows with constructor-specific mixed cells. I then ask whether the same relation geometry can be steered. In an edge-grid clean/corrupt intervention assay over 32 prompts, the row/column scaffold and answer format stay fixed while the YES/NO relation map changes, and the corrupt hidden-state relation frame is patched toward clean or placebo targets. In 70B and 405B, clean-targeted relation-frame paths recover clean-answer behavior and residual relation geometry, while centroid-only and equal-norm controls show negligible recovery. Site/order controls further separate marker-site importance from ordered clean-frame geometry: target clean shape and cross-prompt clean shape recover behavior and residual geometry at the marker interface, whereas corrupt-donor transfer, same-site permutation/reflection, wrong-site clean deltas, centroid-only motion, and equal-norm noise fail or remain far below clean-frame paths. The result is a controlled bridge from relation probing to relation-frame intervention: relation rank geometry can be detected, targeted, and behaviorally validated in transformer hidden states.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑