We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation, outperforming previous methods. Our results indicate that an automated interpretation of neurons can be used for model deception and for introducing new model capabilities.
NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions
Institute for AI Innovation and Societal Impact (AI4LIFE), Hanoi University of Science and Technology(河内国家大学下属科学与技术大学人工智能创新与社会影响研究所);University of Florida(佛罗里达大学);VinUniversity(VinUniversity(越南 VinUniversity 大学))
Interpreting Adversarial Examples by Activation Promotion and Suppression
Northeastern University(东北大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research(IBM研究院)
众所周知,卷积神经网络(CNN)很容易受到对抗性样本的影响:图像带有难以察觉的扰动,用于欺骗分类器。然而,这些扰动的可解释性在文献中较少探讨。这项工作旨在更好地理解对抗性扰动的作用,并从像素,图像和网络的角度提供视觉解释。我们发现,对手对神经元的激活具有促进-抑制效应(PSE),并且可以主要分为三种类型:i)抑制主导的扰动,主要降低真实标签的分类得分,ii)促进主导的扰动,专注于提高目标标签的置信度,以及iii)平衡扰动,在抑制和促进中发挥双重作用。我们还提供了对抗性示例的图像级可解释性。这将像素级扰动的PSE与通过类激活映射定位的类特异性区分图像区域联系起来(Zhou等人,2016)。此外,我们通过网络解剖(Bau et al. 2017)研究了对抗效应,它提供了隐藏单元的概念级可解释性。我们发现,存在着一个紧密的联系之间的单位的敏感性对抗攻击和语义概念上的可解释性。最后,我们从我们的解释中提供了一些新的见解,以提高网络的对抗鲁棒性。
Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models
Technical University of Munich(慕尼黑技术大学);Helmholtz Munich(亥姆霍兹慕尼黑);Munich Center for Machine Learning(慕尼黑机器学习中心);Munich Data Science Institute(慕尼黑数据科学研究所);University of Tübingen(图宾根大学);University of Copenhagen(哥本哈根大学)
Interpreting the linear structure of vision-language model embedding spaces
Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University(哈佛大学自然与人工智能研究所);Department of Computer Science, Harvard University(哈佛大学计算机科学系)