混合层与混合注意力
Mixture of Layers with Hybrid Attention
AI总结：
本文提出MoL，通过减少维度的并行薄块替代全宽度Transformer块，并引入混合注意力解决稀疏路由中的覆盖问题。
摘要：
标准的混合专家（MoE）变压器在每层内将token路由到专家子网络，但层结构本身仍保持单体。我们引入混合层（MoL），用K个并行薄块替代全宽度Transformer块（d_model），通过学习的下/上投影连接，并通过top-k块路由组成。将稀疏块路由扩展到许多块会引发注意力覆盖问题，因为每个块看到的token更少。我们通过引入混合注意力来解决，该机制将一个共享的softmax块用于全局上下文，与路由块中的门控DeltaNet线性注意力相结合。
英文摘要：
Standard Mixture-of-Experts (MoE) transformers route tokens to expert subnetworks within each layer, but the layer structure itself remains monolithic. We introduce Mixture of Layers (MoL), which replaces full-width transformer blocks (d_model) with K parallel thin blocks at reduced dimensionality (d_thin << d_model), connected via learned down/up projections and composed via top-k block routing. Scaling sparse block routing to many blocks creates an attention coverage problem, as each block sees fewer tokens. We address this by introducing hybrid attention, which pairs one shared softmax block for global context with Gated DeltaNet linear attention in routed blocks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
