arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.07706cs.LG
收藏

走向线性的关键:分析驱动的Transformer线性化

The Key to Going Linear: Analysis-Driven Transformer Linearization

Qualcomm AI Research(高通人工智能研究中心)

Anna Kuzina, Paul N. Whatmough, Babak Ehteshami Bejnordi

AI总结:

研究针对因果自注意力二次成本限制长上下文Transformer推理的问题,通过分析状态更新设计,发现softmax原理及近似误差源,引入结构干预,在多模型上扩展线性化方法,提升性能并匹配复杂缓存框架的长上下文检索。

摘要:

因果自注意力的二次成本严重限制了长上下文Transformer推理。虽然存在众多事后线性化管道,但难以确定哪些组件能保持模型质量。本研究在严格的冻结主干机制下分离状态更新设计的影响。表明softmax依赖于与键相关的秩-1正交投影,解释了delta风格网络优于纯门控累加的原因。识别出近似误差的潜在来源并引入结构干预,包括下沉令牌、短卷积和固定预算缓存路由,缩小了剩余差距。在高达32B参数的LLaMA和Qwen模型上扩展此线性化方法,在MMLU上优于先前的事后基线,并与复杂自适应缓存框架的长上下文检索相匹配。

英文摘要:

The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in a strict frozen-backbone regime. We show that softmax relies on key-dependent, rank-1 orthogonal projections, elucidating why delta-style networks outperform purely gated accumulation. We identify a potential source of approximation errors and introduce structural interventions, specifically sink tokens, short convolutions, and fixed-budget cache routing, which reduces the remaining gap. We scale this linearization approach across LLaMA and Qwen models up to 32B parameters, outperforming prior post hoc baselines on MMLU and matching the long-context retrieval of complex adaptive-caching frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑