注意力引导是控制模型聚焦的重要技术,使模型能够实现提示突出,即优先考虑用户指定的文本。然而,现有注意力引导方法需要显式存储完整的注意力矩阵,使其与内存高效的实现如FlashAttention不兼容。我们引入了Spectral Editing Key Amplification (SEKA),一种无需训练的引导方法,通过在注意力计算前直接编辑键嵌入来解决这一问题。SEKA利用谱分解将键嵌入引导至潜在方向,以增强某些令牌的注意力分数。我们扩展了这一方法到Adaptive SEKA (AdaSEKA),一种查询自适应的变体,它使用无需训练的路由机制,根据提示的语义意图动态结合多个专家子空间。我们的实验表明,这两种方法在标准引导基准上显著优于强基线,同时在兼容优化注意力的同时添加了更低的延迟和内存开销。
英文摘要:
Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.
Beyond Prompt: Fine-grained Simulation of Cognitively Impaired Standardized Patients via Stochastic Steering
School of Software, Xi’an Jiaotong University(西安交通大学软件学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究所);College of Computer Science, Sichuan University(四川大学计算机学院);Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China(教育部机器学习与工业智能工程研究中心,中国)
Explaining Intrinsic Moral Self-Correction with Mechanistic Interpretability
Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan(通讯工程研究院,国立台湾大学);MediaTek Research, Taipei, Taiwan(联发科研究,台北,台湾);Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan(电气工程系,国立台湾大学);Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan(电气工程系,国立清华大学);AI Research Center (AINTU), National Taiwan University, Taipei, Taiwan(人工智能研究中心(AINTU),国立台湾大学)
Improving LLM Reasoning through Interpretable Role-Playing Steering
Center for Information and Language Processing, LMU Munich(信息与语言处理中心,慕尼黑大学);Northwestern University(西北大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心(MCML));New Jersey Institute of Technology(新泽西理工学院)
DFKI-MLT at SemEval-2026 TASK 7: Steering Multilingual Models Towards Cultural Knowledge
German Research Center for Artificial Intelligence (DFKI GmbH)(德国人工智能研究中心(DFKI GmbH));Saarland Informatics Campus(萨尔布吕肯信息学校区);Barcelona Supercomputing Center (BSC-CNS)(巴塞罗那超级计算中心(BSC-CNS))