通过学习的标记路由实现自适应计算深度
Adaptive Computation Depth via Learned Token Routing in Transformers
AI总结：
本文提出Token-Selective Attention，通过学习的标记门控机制，在Transformer中动态调整计算深度，减少计算量并提升效率。
摘要：
标准的Transformer架构对每个标记应用相同的层数，无论上下文难度。我们提出Token-Selective Attention（TSA），一种在连续Transformer块之间残差更新上的学习每标记门控。每个门是一个轻量级的两层多层感知机（MLP），产生连续的停止概率，使该机制端到端可微，参数开销为1.7%，不改变基础架构。值得注意的是，TSA在没有显式深度压力的情况下学习难度比例的路由：即使在λ=0（无深度正则化）时，任务损失梯度本身也驱动路由跳过20%的标记层操作。在字符级语言建模中，TSA在Tiny-Shakespeare和enwik8上节省了14-23%的标记层操作（TLOps），在质量损失小于0.5%的情况下。在匹配效率下，TSA的验证损失比早退出低0.7%，且学习的路由直接转移到推理时的稀疏执行，实现实际的时钟速度提升。
英文摘要：
Standard transformer architectures apply the same number of layers to every token regardless of contextual difficulty. We present Token-Selective Attention (TSA), a learned per-token gate on residual updates between consecutive transformer blocks. Each gate is a lightweight two-layer multi-layer perceptron (MLP) that produces a continuous halting probability, making the mechanism end-to-end differentiable with 1.7% parameter overhead and no changes to the base architecture. Notably, TSA learns difficulty-proportional routing without any explicit depth pressure: even at $λ=0$ (no depth regularisation), the task-loss gradient alone drives the router to skip 20% of token-layer operations. On character-level language modeling, TSA saved 14-23% of token-layer operations (TLOps) across Tiny-Shakespeare and enwik8 at <0.5% quality loss. At matched efficiency, TSA achieved 0.7% lower validation loss than early exit, and the learned routing transfers directly to inference-time sparse execution for real wall-clock speedup.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
