arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.12781cs.LGcs.AIcs.CVstat.ML
收藏

输入的相似性是否突出了区别性特征?

Do Input Gradients Highlight Discriminative Features?

Microsoft Research India(微软印度研究院);Google Research India(谷歌印度研究院)

Harshay Shah, Prateek Jain, Praneeth Netrapalli

摘要:

事后基于梯度的可解释性方法[Simonyan等人,2013年,Smilkov等人,2017]提供模型预测的特定实例解释通常基于假设(A):输入梯度的大小-相对于输入的logits梯度-嘈杂地突出了区分性任务相关特征。在这项工作中,我们使用三管齐下的方法来测试假设(A)的有效性。首先,我们开发了一个评估框架,DiffROAR,在四个图像分类基准上测试假设(A)。我们的结果表明,(i)标准模型的输入梯度(即,在原始数据上训练的)可能严重违反(A),而(ii)逆向鲁棒模型的输入梯度满足(A)。其次,我们介绍了BlockMNIST,一个基于MNIST的半真实数据集,通过设计编码的先验知识的判别功能。我们对BlockMNIST的分析利用这些信息来验证和表征标准模型和鲁棒模型的输入梯度属性之间的差异。最后,我们从理论上证明了我们的实证研究结果适用于简化版本的BlockMNIST数据集。具体来说,我们证明了在这个数据集上训练的标准一个隐藏层MLP的输入梯度不会突出特定于实例的信号坐标,因此严重违反了假设(A)。我们的发现激发了以可证伪的方式形式化和测试可解释性中常见假设的需求[Leavitt和Morcos,2020]。我们认为,DiffROAR评估框架和基于BlockMNIST的数据集可以作为健全性检查,以审计特定于实例的可解释性方法;代码和数据可在https://github.com/harshays/inputgradients上获得。

英文摘要:

Post-hoc gradient-based interpretability methods [Simonyan et al., 2013, Smilkov et al., 2017] that provide instance-specific explanations of model predictions are often based on assumption (A): magnitude of input gradients -- gradients of logits with respect to input -- noisily highlight discriminative task-relevant features. In this work, we test the validity of assumption (A) using a three-pronged approach. First, we develop an evaluation framework, DiffROAR, to test assumption (A) on four image classification benchmarks. Our results suggest that (i) input gradients of standard models (i.e., trained on original data) may grossly violate (A), whereas (ii) input gradients of adversarially robust models satisfy (A). Second, we introduce BlockMNIST, an MNIST-based semi-real dataset, that by design encodes a priori knowledge of discriminative features. Our analysis on BlockMNIST leverages this information to validate as well as characterize differences between input gradient attributions of standard and robust models. Finally, we theoretically prove that our empirical findings hold on a simplified version of the BlockMNIST dataset. Specifically, we prove that input gradients of standard one-hidden-layer MLPs trained on this dataset do not highlight instance-specific signal coordinates, thus grossly violating assumption (A). Our findings motivate the need to formalize and test common assumptions in interpretability in a falsifiable manner [Leavitt and Morcos, 2020]. We believe that the DiffROAR evaluation framework and BlockMNIST-based datasets can serve as sanity checks to audit instance-specific interpretability methods; code and data available at https://github.com/harshays/inputgradients.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑