SCoCCA: 通过典型相关分析进行多模态稀疏概念分解
SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis
AI总结：
本文提出SCoCCA，通过典型相关分析实现多模态稀疏概念分解，提升跨模态对齐与可解释性，实现概念发现的最优性能。
摘要：
解释视觉-语言模型的内部推理对于在安全关键领域部署AI至关重要。基于概念的可解释性通过语义有意义的组件表示模型行为。然而，现有方法主要局限于图像，忽略了跨模态交互。文本-图像嵌入，如CLIP生成的嵌入，存在模态间隙，视觉和文本特征遵循不同的分布，限制了可解释性。典型相关分析（CCA）提供了一种原则性方法来对齐不同分布的特征，但尚未用于多模态概念层面分析。我们证明CCA和InfoNCE的目标密切相关，优化CCA隐含优化InfoNCE，提供一种简单、无需训练的机制来增强跨模态对齐而不影响预训练的InfoNCE目标。受此启发，我们结合基于概念的可解释性与CCA，引入概念CCA（CoCCA），一种对齐跨模态嵌入并实现可解释的概念分解的框架。我们进一步扩展并提出稀疏概念CCA（SCoCCA），强制稀疏性以产生更解耦和判别性的概念，促进改进的激活、消融和语义操作。我们的方法将基于概念的解释扩展到多模态嵌入，并在概念发现中实现最先进的性能，证据来自重建和操作任务如概念消融。
英文摘要：
Interpreting the internal reasoning of vision-language models is essential for deploying AI in safety-critical domains. Concept-based explainability provides a human-aligned lens by representing a model's behavior through semantically meaningful components. However, existing methods are largely restricted to images and overlook the cross-modal interactions. Text-image embeddings, such as those produced by CLIP, suffer from a modality gap, where visual and textual features follow distinct distributions, limiting interpretability. Canonical Correlation Analysis (CCA) offers a principled way to align features from different distributions, but has not been leveraged for multi-modal concept-level analysis. We show that the objectives of CCA and InfoNCE are closely related, such that optimizing CCA implicitly optimizes InfoNCE, providing a simple, training-free mechanism to enhance cross-modal alignment without affecting the pre-trained InfoNCE objective. Motivated by this observation, we couple concept-based explainability with CCA, introducing Concept CCA (CoCCA), a framework that aligns cross-modal embeddings while enabling interpretable concept decomposition. We further extend it and propose Sparse Concept CCA (SCoCCA), which enforces sparsity to produce more disentangled and discriminative concepts, facilitating improved activation, ablation, and semantic manipulation. Our approach generalizes concept-based explanations to multi-modal embeddings and achieves state-of-the-art performance in concept discovery, evidenced by reconstruction and manipulation tasks such as concept ablation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
