因果提示校准引导的 Segment Anything Model 用于开放词汇多实体分割
Causal Prompt Calibration Guided Segment Anything Model for Open-Vocabulary Multi-Entity Segmentation
AI总结：
针对 SAM 在开放词汇多实体分割中的提示偏差问题，提出 CPC-SAM，通过 CaPL 学习去混杂的因果提示并以双层优化实现一致且最优的分割。
摘要：
尽管 Segment Anything Model（SAM）具有强大能力，但它在开放词汇多实体分割（OVMS）中仍面临泛化问题。通过实证分析和因果分析，我们发现：（i）提示偏差是泛化问题的主要原因；（ii）这种偏差与提示中与任务无关的生成因素密切相关，这些因素充当混杂因素并影响泛化。为解决泛化问题，我们旨在提出一种能够校准提示以消除混杂因素、从而实现准确 OVMS 的方法。在因果分析的基础上，我们提出 OVMS 的最优提示应只包含与任务相关的因果因素。我们将其定义为因果提示，并将其作为校准目标。随后，我们以因果多分布一致性理论为基础进行理论分析，证明可以通过强制分割一致性和最优性来获得该提示。受此启发，我们提出 CPC-SAM，这是一种面向 SAM 的因果提示校准方法，用于实现准确的 OVMS。该方法将轻量级因果提示学习器（CaPL）集成到 SAM 中以获得因果提示。具体而言，我们首先使用随机标注生成多个提示以模拟不同分布，然后通过 CaPL 在任务层和实体层强制因果多分布一致性，对这些提示重新加权。为确保获得因果提示，通过最小化重加权提示上的累积分割损失来优化 CaPL，以实现一致性和最优性。双层优化策略在优化 CaPL 与 SAM 之间交替进行，从而确保准确的 OVMS。大量实验验证了该方法的优越性。
英文摘要：
Despite the strength of the Segment Anything Model (SAM), it struggles with generalization issues in open-vocabulary multi-entity segmentation (OVMS). Through empirical and causal analyses, we find that (i) the prompt bias is the primary cause of the generalization issues; (ii) this bias is closely tied to the task-irrelevant generating factors within the prompts, which act as confounders and affect generalization. To address the generalization issues, we aim to propose a method that can calibrate prompts to eliminate confounders for accurate OVMS. Building upon the causal analysis, we propose that the optimal prompt for OVMS should contain only task-relevant causal factors. We define it as the causal prompt, serving as the goal of calibration. Next, our theoretical analysis, grounded by causal multi-distribution consistency theory, proves that this prompt can be obtained by enforcing segmentation consistency and optimality. Inspired by this, we propose CPC-SAM, a Causal Prompt Calibration method for SAM to achieve accurate OVMS. It integrates a lightweight causal prompt learner (CaPL) into SAM to obtain causal prompts. Specifically, we first generate multiple prompts using random annotations to simulate diverse distributions and then reweight them via CaPL by enforcing causal multi-distribution consistency in both task and entity levels. To ensure obtaining causal prompts, CaPL is optimized by minimizing the cumulative segmentation loss across the reweighted prompts to achieve consistency and optimality. A bi-level optimization strategy alternates between optimizing CaPL and SAM, ensuring accurate OVMS. Extensive experiments validate its superiority.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
