说的比做的更容易：伪造线性解码方法的因果关系
Much Easier Said Than Done: Falsifying the Causal Relevance of Linear Decoding Methods
摘要：
线性分类器探测器经常用于更好地理解神经网络的功能。研究人员已经通过探索神经网络的学习和内部表征来解决确定单元重要性的问题。线性分类器探测器将高度选择性的单元识别为对网络功能最重要的单元。网络是否实际上依赖于高选择性单元，可以通过使用消融将它们从网络中移除来测试。令人惊讶的是，当高度选择性的单元被消融时，它们只会产生很小的性能缺陷，而且即使如此，也只是在某些情况下。尽管没有选择性神经元的消融效应，但线性解码方法可以有效地用于解释网络功能，使其有效性成为一个谜。为了证明选择性在网络功能中的独特作用并解决这一矛盾，我们系统地消融了激活空间子区域中的单元群。在这里，我们发现探针识别的神经元和消融识别的神经元之间的关系很弱。更具体地说，我们发现，选择性和单位的平均活动之间的相互作用更好地预测消融性能赤字组的单位在AlexNet，VGG16，MobileNetV2和ResNet101。线性解码器可能有点有效，因为它们与那些对网络功能有因果关系的单元重叠。可解释性方法可以通过关注因果重要性单元来改进。
英文摘要：
Linear classifier probes are frequently utilized to better understand how neural networks function. Researchers have approached the problem of determining unit importance in neural networks by probing their learned, internal representations. Linear classifier probes identify highly selective units as the most important for network function. Whether or not a network actually relies on high selectivity units can be tested by removing them from the network using ablation. Surprisingly, when highly selective units are ablated they only produce small performance deficits, and even then only in some cases. In spite of the absence of ablation effects for selective neurons, linear decoding methods can be effectively used to interpret network function, leaving their effectiveness a mystery. To falsify the exclusive role of selectivity in network function and resolve this contradiction, we systematically ablate groups of units in subregions of activation space. Here, we find a weak relationship between neurons identified by probes and those identified by ablation. More specifically, we find that an interaction between selectivity and the average activity of the unit better predicts ablation performance deficits for groups of units in AlexNet, VGG16, MobileNetV2, and ResNet101. Linear decoders are likely somewhat effective because they overlap with those units that are causally important for network function. Interpretability methods could be improved by focusing on causally important units.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
