arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.01281cs.CLcs.AI
收藏

光谱注意力引导用于提示突出

Spectral Attention Steering for Prompt Highlighting

University of Edinburgh, UK(爱丁堡大学);RayNeo, China(RayNeo);Huawei Research Ltd., UK(华为研究有限公司);Queen Mary University of London, UK(伦敦大学玛丽女王学院)

Weixian Waylon Li, Yuchen Niu, Yongxin Yang, Keshuang Li, Tiejun Ma, Shay B. Cohen

AI总结:

SEKA和AdaSEKA通过直接编辑键嵌入实现无需训练的注意力引导,显著提升提示突出性能,同时保持低延迟和内存开销。

摘要:

注意力引导是控制模型聚焦的重要技术,使模型能够实现提示突出,即优先考虑用户指定的文本。然而,现有注意力引导方法需要显式存储完整的注意力矩阵,使其与内存高效的实现如FlashAttention不兼容。我们引入了Spectral Editing Key Amplification (SEKA),一种无需训练的引导方法,通过在注意力计算前直接编辑键嵌入来解决这一问题。SEKA利用谱分解将键嵌入引导至潜在方向,以增强某些令牌的注意力分数。我们扩展了这一方法到Adaptive SEKA (AdaSEKA),一种查询自适应的变体,它使用无需训练的路由机制,根据提示的语义意图动态结合多个专家子空间。我们的实验表明,这两种方法在标准引导基准上显著优于强基线,同时在兼容优化注意力的同时添加了更低的延迟和内存开销。

英文摘要:

Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑