arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.10705cs.CL
收藏

Prism-$Δ$:用于大语言模型中快速突出的差分子空间操控

PRISM-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models

Yuyao Ge, Shenghua Liu, Yiwei Wang, Baolong Bi, Lingrui Mei, Jiayu Yao, Jiafeng Guo, Xueqi Cheng

AI总结:

PRISM-$Δ$通过差分子空间操控提升大语言模型的提示突出效果,实现更高的生成质量与更低的资源消耗。

摘要:

提示突出引导大型语言模型在生成过程中优先考虑用户指定的文本片段。一个关键挑战是提取能够捕捉相关与无关上下文之间差异的引导方向,而不是两者共有的结构模式。我们提出了PRISM-$Δ$(基于投影的相关性引导方法),该方法通过分解正负交叉协方差矩阵的差异,以最大化判别能量并消除共享方向。每个注意力头接收一个连续的softplus重要性权重,使弱但有用的头在减弱的强度下做出贡献。该框架自然扩展到价值表示,捕捉到键-only方法未使用的内容通道信号。在四个基准和五个模型上,PRISM-$Δ$在19种配置中匹配或超过了现有最佳方法,相对增益高达+10.6%,同时将引导的流畅度成本减半。PRISM-$Δ$还扩展到长上下文检索,在此方面,它比现有最佳方法高出高达+4.8%的相对增益。PRISM-$Δ$与FlashAttention兼容,并且增加了极小的内存开销。

英文摘要:

Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge of existing Key-editing approaches is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-$Δ$ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, our methods match or exceed the best existing method on 19 of 20 configurations, with relative gains up to +13.0%, while halving the fluency cost of steering. Our methods also scale to long-context retrieval, outperforming the best existing method by up to +5.1% relative gain. PRISM-$Δ$ is compatible with FlashAttention and adds negligible memory overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑