向混合注意力模型演进的策略内方法
On-Policy Morphing into Hybrid Attention Models
AI总结:
本研究针对混合注意力模型转换的层选择问题,提出策略内方法FlashMorph2,通过端到端评估离散混合架构、联合优化层选择分数与模型权重,在Qwen3模型上较FlashMorph提升性能且效率更高。
摘要:
混合注意力模型结合全注意力与线性注意力,以实现更优的质量-效率权衡。将预训练Transformer转换为这类模型,需在规定预算下确定哪些层保留全注意力。我们发现FlashMorph的层选择存在两个策略外不匹配问题:(i)连续分支插值与教师强制隐藏状态不同于离散混合计算;(ii)冻结模型权重未考虑后续恢复过程中的参数适配。我们提出FlashMorph2,这是一种策略内的层选择方法:(I)使用采样的离散混合架构自身的隐藏状态对其进行端到端评估;(II)在恢复目标下联合优化层选择分数与模型权重,以反映适配后的性能。在Qwen3模型规模及线性注意力变体上的实验显示,FlashMorph2始终优于现有层选择方法,在RULER上较FlashMorph平均分数提升2.42个点,在LongBench上提升1.08个点,在高召回任务上提升0.76个点,且在0.6B至8B模型上完成层选择仅需0.72至3.57个GPU小时,展现出高效率与可扩展性。
英文摘要:
Hybrid attention models combine full and linear attention to achieve a better quality--efficiency trade-off. Converting pretrained Transformers into such models requires identifying which layers should retain full attention under a prescribed budget. We identify two off-policy mismatches in FlashMorph's layer selection: (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery. We introduce FlashMorph2, an on-policy layer-selection method that (I) evaluates sampled discrete hybrid architectures end to end using their own hidden states, and (II) jointly optimizes layer-selection scores and model weights under the recovery objective to reflect post-adaptation performance. Experiments across Qwen3 model scales and linear-attention variants show that FlashMorph2 consistently outperforms existing layer-selection methods, improving average scores over FlashMorph by $+2.42$ points on RULER, $+1.08$ on LongBench, and $+0.76$ on recall-intensive tasks, while completing layer selection in only $0.72$--$3.57$ GPU hours across 0.6B--8B models, demonstrating high efficiency and scalability.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。