基于概念引导的微调：引导ViT远离虚假相关性以提高鲁棒性
Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness
AI总结：
本文提出一种引导模型推理至概念层面语义的微调框架，通过优化内部相关性映射对齐空间 grounded 的概念掩码，提升模型在分布偏移下的鲁棒性。
摘要：
视觉变换器（ViTs）在分布偏移下往往性能下降，因为它们依赖于虚假相关性，如背景提示，而非语义性的特征。现有的正则化方法通常依赖于简单的前景-背景掩码，无法捕捉定义对象的细粒度语义概念（例如，``长喙''和``翅膀''对于``鸟''）。因此，这些方法对分布偏移的鲁棒性有限。为了解决这一限制，我们引入了一种新的微调框架，引导模型推理至概念层面语义。我们的方法优化模型内部的相关性映射，使其与空间上接地的概念掩码对齐。这些掩码是自动产生的，无需人工标注：首先使用基于LLM的无标签方法提出类相关概念，然后通过视觉语言模型进行分割。微调目标将相关性与这些概念区域对齐，同时抑制对虚假背景区域的关注。值得注意的是，这一过程只需要少量图像，并使用了半数的数据集类别。在五个分布外基准测试中，我们的方法在多个基于ViT的模型上提升了鲁棒性。此外，我们展示了生成的相关性映射在语义对象部分上表现出更强的对齐，为更稳健和可解释的视觉模型提供了可扩展的路径。最后，我们确认概念引导的掩码比传统分割地图更能有效监督模型的鲁棒性，支持我们的核心假设。
英文摘要：
Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations, such as background cues, rather than semantically meaningful features. Existing regularization methods, typically relying on simple foreground-background masks, which fail to capture the fine-grained semantic concepts that define an object (e.g., ``long beak'' and ``wings'' for a ``bird''). As a result, these methods provide limited robustness to distribution shifts. To address this limitation, we introduce a novel finetuning framework that steers model reasoning toward concept-level semantics. Our approach optimizes the model's internal relevance maps to align with spatially grounded concept masks. These masks are generated automatically, without manual annotation: class-relevant concepts are first proposed using an LLM-based, label-free method, and then segmented using a VLM. The finetuning objective aligns relevance with these concept regions while simultaneously suppressing focus on spurious background areas. Notably, this process requires only a minimal set of images and uses half of the dataset classes. Extensive experiments on five out-of-distribution benchmarks demonstrate that our method improves robustness across multiple ViT-based models. Furthermore, we show that the resulting relevance maps exhibit stronger alignment with semantic object parts, offering a scalable path toward more robust and interpretable vision models. Finally, we confirm that concept-guided masks provide more effective supervision for model robustness than conventional segmentation maps, supporting our central hypothesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
