arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.04341cs.CV
收藏

解读CLIP中神经元的二阶效应

Interpreting the Second-Order Effects of Neurons in CLIP

UC Berkeley(加州大学伯克利分校)

Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt

AI总结:

针对CLIP神经元功能解读难题,提出“二阶透镜”分析其通过后续注意力头直接流向输出的效应,揭示多义性并用于批量生成语义对抗样本与零样本分割,性能更优且拓展模型能力。

摘要:

我们通过使用文本自动描述CLIP中的单个神经元来解读其功能。分析直接效应(即神经元通过残差流到输出的信息流)或间接效应(整体贡献)无法捕捉CLIP中神经元的功能。因此,我们提出“二阶透镜”,分析从神经元通过后续注意力头直接流向输出的效应。我们发现这些效应具有高度选择性:对于每个神经元,其效应仅对<2%的图像显著。此外,每个效应可通过CLIP的文本-图像空间中的单一方向近似。我们通过将这些方向分解为稀疏的文本表示集来描述神经元。这些集合揭示了多义行为——每个神经元对应多个通常不相关的概念(例如船舶和汽车)。利用这种神经元多义性,我们通过生成与错误类别虚假相关的概念的图像来批量生产“语义”对抗样本。此外,我们将二阶效应用于零样本分割,性能优于先前方法。我们的结果表明,神经元的自动解读可用于模型欺骗和引入新的模型能力。

英文摘要:

We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation, outperforming previous methods. Our results indicate that an automated interpretation of neurons can be used for model deception and for introducing new model capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑