评估稀疏自编码器中概念表示的对抗鲁棒性
Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders
AI总结：
本研究评估了稀疏自编码器中概念表示的对抗鲁棒性，发现其易受微小输入扰动影响，可能不适合用于模型监控和监督。
摘要：
稀疏自编码器（SAEs）常用于通过将它们映射到人类可解释的概念表示来解释大型语言模型（LLMs）的内部激活。尽管现有对SAEs的评估主要关注诸如重建-稀疏性权衡、人类（自动）可解释性和特征解耦等指标，但它们忽略了关键方面：概念表示对输入扰动的鲁棒性。我们认为鲁棒性必须是概念表示的基本考虑因素，反映概念标记的准确性。为此，我们将鲁棒性量化 formulation 为输入空间优化问题，并开发了一个综合的评估框架，其中包含现实场景，其中对抗扰动被设计用来操纵SAE表示。实证上，我们发现微小的对抗输入扰动可以在大多数情况下有效操纵基于概念的解释，而不会显著影响基础LLM的激活。总体而言，我们的结果表明，SAE概念表示是脆弱的，且在未进一步去噪或后处理的情况下，可能不适合用于模型监控和监督应用。
英文摘要：
Sparse autoencoders (SAEs) are commonly used to interpret the internal activations of large language models (LLMs) by mapping them to human-interpretable concept representations. While existing evaluations of SAEs focus on metrics such as the reconstruction-sparsity tradeoff, human (auto-)interpretability, and feature disentanglement, they overlook a critical aspect: the robustness of concept representations to input perturbations. We argue that robustness must be a fundamental consideration for concept representations, reflecting the fidelity of concept labeling. To this end, we formulate robustness quantification as input-space optimization problems and develop a comprehensive evaluation framework featuring realistic scenarios in which adversarial perturbations are crafted to manipulate SAE representations. Empirically, we find that tiny adversarial input perturbations can effectively manipulate concept-based interpretations in most scenarios without notably affecting the base LLM's activations. Overall, our results suggest that SAE concept representations are fragile and without further denoising or postprocessing they might be ill-suited for applications in model monitoring and oversight.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
