探针分类器在概念移除与检测中不可靠
Probing Classifiers are Unreliable for Concept Removal and Detection
AI总结:
本文通过理论与实验证明,依赖探针分类器的事后或对抗性概念移除方法无法彻底移除概念,甚至可能破坏任务特征,并提出虚假相关性指标以评估最终分类器质量。
摘要:
针对文本数据训练的神经网络模型已被发现会在其表示中编码不良的语言或敏感概念。由于概念、文本输入和学习到的表示之间存在复杂关系,移除这些概念并非易事。近期工作提出了事后(post-hoc)和对抗性(adversarial)方法来从模型表示中移除这些不需要的概念。通过广泛的理论和实证分析,我们表明这些方法可能适得其反:它们无法完全移除概念,在最坏情况下可能最终破坏所有与任务相关的特征。原因在于这些方法依赖探针分类器(probing classifier)作为概念的代理。即使在最有利于学习探针分类器的条件下——即概念在表示空间中的相关特征单独就能提供100%准确率——我们证明探针分类器仍可能使用非概念特征,因此事后或对抗性方法将无法正确移除概念。这些理论推论通过在合成数据集、Multi-NLI和Twitter数据集上训练的模型实验得到证实。对于概念移除的敏感应用(如公平性),我们建议谨慎使用这些方法,并提出一个虚假相关性(spuriousness)指标来衡量最终分类器的质量。
英文摘要:
Neural network models trained on text data have been found to encode undesirable linguistic or sensitive concepts in their representation. Removing such concepts is non-trivial because of a complex relationship between the concept, text input, and the learnt representation. Recent work has proposed post-hoc and adversarial methods to remove such unwanted concepts from a model's representation. Through an extensive theoretical and empirical analysis, we show that these methods can be counter-productive: they are unable to remove the concepts entirely, and in the worst case may end up destroying all task-relevant features. The reason is the methods' reliance on a probing classifier as a proxy for the concept. Even under the most favorable conditions for learning a probing classifier when a concept's relevant features in representation space alone can provide 100% accuracy, we prove that a probing classifier is likely to use non-concept features and thus post-hoc or adversarial methods will fail to remove the concept correctly. These theoretical implications are confirmed by experiments on models trained on synthetic, Multi-NLI, and Twitter datasets. For sensitive applications of concept removal such as fairness, we recommend caution against using these methods and propose a spuriousness metric to gauge the quality of the final classifier.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。