基于受控保留信息的仅解码器LLM归因忠实性评估
Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Information
AI总结：
针对现有软扰动忠实性指标因保留词数不同导致评估偏差的问题，提出π-Soft-NC和π-Soft-NS框架，通过控制期望保留概率公平比较归因方法，并引入专用于自回归解码器LLM的梯度归因方法Grad-ELLM。
摘要：
大型语言模型（LLM）越来越多地使用输入归因方法进行评估，但比较这些解释仍然具有挑战性。现有的软扰动忠实性指标，如Soft-NC和Soft-NS，可能将归因质量与扰动期间保留的词数混为一谈：平均得分较高的归因方法可能保留更多词，从而获得膨胀的分数。为解决此问题，我们提出π-Soft-NC和π-Soft-NS，这是一个在相同期望保留概率下比较归因方法的评估框架，从而控制保留词数。我们进一步引入Grad-ELLM，一种针对自回归仅解码器LLM定制的基于梯度的归因方法，该方法在每个解码步骤将梯度导出的通道重要性与注意力导出的标记重要性相结合。在Llama和Mistral上的分类和开放生成任务实验表明，Grad-ELLM在π-Soft-NC下实现了强全面性导向的忠实性，而在π-Soft-NS下没有主导方法。我们的评估指标为比较LLM的可解释人工智能方法提供了一个严格的框架，将支持该领域的进展。
英文摘要：
Large Language Models (LLMs) are increasingly evaluated with input attribution methods, yet comparing such explanations remains challenging. Existing soft-perturbation faithfulness metrics, such as Soft-NC and Soft-NS, can conflate attribution quality with the number of words retained during perturbation: attribution methods with larger average scores may keep more words and therefore obtain inflated scores. To address this issue, we propose $π$-Soft-NC and $π$-Soft-NS, an evaluation framework that compares attribution methods under the same expected retaining probability, thus controlling the number of retained words. We further introduce Grad-ELLM, a gradient-based attribution method tailored to autoregressive decoder-only LLMs, which combines gradient-derived channel importance with attention-derived token importance at each decoding step. Experiments on classification and open-generation tasks with Llama and Mistral show that Grad-ELLM achieves strong comprehensiveness-oriented faithfulness under $π$-Soft-NC, while there is no dominant method under $π$-Soft-NS. Our evaluation metric serves as a rigorous framework to compare XAI methods for LLMs, which will support progress in the field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
