DTRNet：降低Transformer二次方计算开销的动态令牌路由网络
DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers
AI总结：
针对Transformer自注意力二次方计算成本高的问题，提出DTRNet动态令牌路由架构，让多数令牌仅做线性更新、仅约10%令牌走注意力，在同等FLOPs下准确率和内存表现优于MoD等方法，长序列效率提升更显著。
摘要：
Transformer在众多任务上取得了当前最优结果，但它们在每一层对每个令牌统一应用二次方复杂度的自注意力机制，导致计算成本高昂。我们提出了DTRNet（动态令牌路由网络），这是一种改进的Transformer架构，允许令牌动态跳过跨令牌混合的二次方开销，同时仍能接收轻量级的线性更新。通过保留MLP模块并将大多数令牌的注意力开销降至线性级别，DTRNet确保每个令牌都得到显式更新，同时显著降低整体计算量。该设计为标准密集注意力提供了一种高效且有效的替代方案。训练完成后，DTRNet模块在每一层仅将约10%的令牌路由通过注意力机制，同时保持与完整Transformer相当的性能。在相同浮点运算量（FLOPs）下，它在准确率和内存表现上始终优于基于路由的层跳过方法（如MoD和D-LLM），且路由至全注意力的令牌数量更少。其效率增益随序列长度提升，针对长上下文输入可大幅减少FLOPs。通过将令牌更新与注意力混合解耦，DTRNet大幅降低了计算中的二次方占比，为Transformer提供了一种简单、高效且可扩展的替代方案。
英文摘要：
Transformers achieve state-of-the-art results across many tasks, but their uniform application of quadratic self-attention to every token at every layer makes them computationally expensive. We introduce DTRNet (Dynamic Token Routing Network), an improved Transformer architecture that allows tokens to dynamically skip the quadratic cost of cross-token mixing while still receiving lightweight linear updates. By preserving the MLP module and reducing the attention cost for most tokens to linear, DTRNet ensures that every token is explicitly updated while significantly lowering overall computation. This design offers an efficient and effective alternative to standard dense attention. Once trained, DTRNet blocks routes only ~10% of tokens through attention at each layer while maintaining performance comparable to a full Transformer. It consistently outperforms routing-based layer skipping methods such as MoD and D-LLM in both accuracy and memory at matched FLOPs, while routing fewer tokens to full attention. Its efficiency gains, scales with sequence length, offering significant reduction in FLOPs for long-context inputs. By decoupling token updates from attention mixing, DTRNet substantially reduces the quadratic share of computation, providing a simple, efficient, and scalable alternative to Transformers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
