arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.14923cs.LGcs.AI
收藏

Transformer中的方向路由

Directional Routing in Transformers

Kevin Taylor

AI总结:

本文提出了一种轻量级机制,通过共享路由器控制每个transformer注意力头的抑制方向,降低参数成本,并展示了路由在模型中的核心作用及对性能的影响。

摘要:

我们引入了方向路由,这是一种轻量级机制,通过共享路由器控制每个transformer注意力头的学习抑制方向,参数成本仅为3.9%。我们在一个运行中训练了一个433000000参数模型及其相同的基线模型,然后通过机械可解释性追踪所得到的电路。路由成为模型的主要计算路径。禁用它会使事实回忆的概率在所有8个测试提示中降至接近零,归纳准确率从93.4%降至0.0%。禁用单个注意力头的影响微乎其微:主要移动头的移除实际上增加了目标概率,而归纳头在没有其最强成员的情况下仍保持98.6%的准确率。协调机制是不可替代的;它所协调的组件也不是。模型还自我组织,没有显式压力,分成两个领域:早期层中的领域适应路由和后期层中的固定句法修剪,其中变化最少的层是最关键的(禁用时PPL增加42.6)。路由将困惑度降低31-56%相对基线,尽管下游多项选择基准目前尚未反映这些收益。

英文摘要:

We introduce directional routing, a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router, at 3.9% parameter cost. We train a 433M-parameter model alongside an identical baseline in a single run, then trace the resulting circuits through mechanistic interpretability. Routing becomes the model's dominant computational pathway. Disabling it collapses factual recall to near-zero probability across all 8 test prompts and drops induction accuracy from 93.4% to 0.0%. Knocking out individual attention heads has negligible effect: the primary mover head's removal actually increases target probability, and induction heads retain 98.6% accuracy without their strongest member. The coordination mechanism is irreplaceable; the components it coordinates are not. The model also self-organizes, without explicit pressure, into two regimes: domain-adaptive routing in early layers and fixed syntactic pruning in late layers, where the least-varying layer is the most critical (+42.6 PPL when disabled). Routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑