RoPE-LIME: RoPE空间局部性 + 稀疏K采样用于高效的LLM归因
RoPE-LIME: RoPE-Space Locality + Sparse-K Sampling for Efficient LLM Attribution
AI总结:
RoPE-LIME通过RoPE空间局部性和稀疏K采样,高效实现LLM归因,减少API调用并提升归因信息质量。
摘要:
解释封闭源大型语言模型(LLM)的输出具有挑战性,因为API访问阻止了基于梯度的归因,而扰动方法在依赖再生文本时成本高且噪声大。我们引入了RoPE-LIME,即Rotary Positional Embedding Linear Local Interpretable Model-agnostic Explanations,这是gSMILE的开源扩展,将推理与解释分离:给定封闭模型的固定输出,一个较小的开源替代模型从基于概率的目标(负对数似然和发散目标)下通过输入扰动计算token级归因。RoPE-LIME结合(i)基于Relaxed Word Mover's Distance计算的局部核,该距离在RoPE嵌入空间中计算以在掩码下保持稳定的相似性,以及(ii)稀疏K采样,一种高效的扰动策略,在有限预算下提高交互覆盖范围。在HotpotQA(句子特征)和手动标注的MMLU子集(词特征)上的实验表明,RoPE-LIME产生的归因信息比leave-one-out采样更丰富,并在显著减少封闭模型API调用的同时优于gSMILE。
英文摘要:
Explaining closed-source Large Language Model (LLM) outputs is challenging because API access prevents gradient-based attribution, while perturbation methods are costly and noisy when they depend on regenerated text. We introduce \textbf{Rotary Positional Embedding Linear Local Interpretable Model-agnostic Explanations (RoPE-LIME)}, an open-source extension of gSMILE that decouples reasoning from explanation: given a fixed output from a closed model, a smaller open-source surrogate computes token-level attributions from probability-based objectives (negative log-likelihood and divergence targets) under input perturbations. RoPE-LIME incorporates (i) a locality kernel based on Relaxed Word Mover's Distance computed in \textbf{RoPE embedding space} for stable similarity under masking, and (ii) \textbf{Sparse-$K$} sampling, an efficient perturbation strategy that improves interaction coverage under limited budgets. Experiments on HotpotQA (sentence features) and a hand-labeled MMLU subset (word features) show that RoPE-LIME produces more informative attributions than leave-one-out sampling and improves over gSMILE while substantially reducing closed-model API calls.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。