arXiv 2006.091282020-06cs.LGcs.CVstat.ML

重新思考基于行为的归因方法对模型可解释性的作用

Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability

Idiap Research Institute（伊迪亚普研究所）；EPFL（洛桑联邦理工学院）；University of Geneva（日内瓦大学）

Suraj Srinivas, Francois Fleuret

备注：: Oral Presentation at ICLR 2021

摘要：

当前用于区分性深度神经网络的可解释性的方法通常依赖于模型的输入梯度，即，输出logit的梯度w.r.t.输入。常见的假设是，这些输入梯度包含有关$p_θ ( y \mid x)$的信息，模型的判别能力，从而证明他们的使用可解释性。然而，在这项工作中，我们表明，这些输入梯度可以任意操纵的结果softmax的平移不变性，而不改变判别函数。这留下了一个悬而未决的问题：如果输入梯度可以是任意的，为什么它们在标准模型中是高度结构化和解释性的？我们调查这一点，通过重新解释标准的softmax为基础的分类器的非归一化的对数密度的数据分布，并表明输入梯度可以被视为梯度的类条件密度模型$p_θ(x \mid y)$内隐含的歧视性模型。这使我们假设，输入梯度的高度结构化和解释性的性质可能是由于这种类条件模型$p_θ(x \mid y)$与地面实况数据分布$p_{\text{data}} (x \mid y)$的对齐。我们通过研究密度排列对梯度解释的影响来检验这一假设。为了实现这种对齐，我们使用分数匹配，并提出了新的近似算法，使训练大规模的模型。我们的实验表明，改进隐式密度模型与数据分布的对齐增强了梯度结构和解释能力，而减少这种对齐具有相反的效果。总的来说，我们发现输入梯度捕获了关于隐式生成模型的信息，这意味着我们需要重新思考它们在解释判别模型时的用途。

英文摘要：

Current methods for the interpretability of discriminative deep neural networks commonly rely on the model's input-gradients, i.e., the gradients of the output logits w.r.t. the inputs. The common assumption is that these input-gradients contain information regarding $p_θ ( y \mid x)$, the model's discriminative capabilities, thus justifying their use for interpretability. However, in this work we show that these input-gradients can be arbitrarily manipulated as a consequence of the shift-invariance of softmax without changing the discriminative function. This leaves an open question: if input-gradients can be arbitrary, why are they highly structured and explanatory in standard models? We investigate this by re-interpreting the logits of standard softmax-based classifiers as unnormalized log-densities of the data distribution and show that input-gradients can be viewed as gradients of a class-conditional density model $p_θ(x \mid y)$ implicit within the discriminative model. This leads us to hypothesize that the highly structured and explanatory nature of input-gradients may be due to the alignment of this class-conditional model $p_θ(x \mid y)$ with that of the ground truth data distribution $p_{\text{data}} (x \mid y)$. We test this hypothesis by studying the effect of density alignment on gradient explanations. To achieve this alignment we use score-matching, and propose novel approximations to this algorithm to enable training large-scale models. Our experiments show that improving the alignment of the implicit density model with the data distribution enhances gradient structure and explanatory power while reducing this alignment has the opposite effect. Overall, our finding that input-gradients capture information regarding an implicit generative model implies that we need to re-think their use for interpreting discriminative models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文