大语言模型中用于对比解码的注意力引导层选择
Attention-Guided Layer Selection for Contrastive Decoding in Large Language Models
浏览论文内容
中文总结 AI 辅助
研究大语言模型中对比解码的层选择问题，提出Attention-JSD等三种基于注意力引导的策略，实验表明这些策略在TruthfulQA上优于原始DoLa，在多答案指标上有显著提升，凸显注意力分布在解决事实性知识上的优势。
中文摘要 AI 辅助
对比解码方法如DoLa通过对比成熟层和未成熟层的输出分布来提高大语言模型（LLMs）的事实性。然而，DoLa的动态层选择仅依赖于输出词汇分布的差异。在这项工作中，我们提出了三种注意力引导策略：Attention-JSD、Attention-Entropy-Max和Attention-Entropy-Min，利用内部自注意力机制携带的结构信息作为层选择的信号。在TruthfulQA上的实验结果表明，我们的策略，特别是Attention-JSD和Attention-Entropy-Min，始终优于原始的DoLa。我们在多答案指标（MC2和MC3）上观察到显著提升，表明注意力分布比输出词汇分布能提供更敏感的信号来解决事实性知识。
英文摘要
Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and premature layers. However, DoLa's dynamic layer selection relies solely on divergences in output vocabulary distributions. In this work, we propose three attention-guided strategies: Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min, which leverage structural information carried by internal self-attention mechanisms as a signal for layer selection. Experimental results on TruthfulQA demonstrate that our strategies, particularly Attention-JSD and Attention-Entropy-Min, consistently outperform the original DoLa. We observe significant gains on multi-answer metrics (MC2 and MC3), suggesting that attention distributions can provide a more sensitive signal for resolving factual knowledge than output vocabulary distributions.
发表机构
- Japan Advanced Institute of Science and Technology（日本先进科学技术学院）
机构由 AI 辅助整理，请以论文原文为准。
