通过基于扩散的反事实生成实现因果对齐的提示学习
Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation
AI总结：
本文提出 DiCap，利用基于扩散的反事实生成与对比学习提取因果对齐、可跨类别泛化的提示，并在多类视觉任务上验证其优势。
摘要：
提示学习因其相较于传统模型训练和微调的高效性而受到关注。然而，现有方法受限于不充分的理论基础，在获得因果不变提示方面遇到困难，最终无法捕捉可在各类别间有效泛化的鲁棒特征。为应对这些挑战，我们提出 DiCap 模型，这是一个具有理论依据、基于扩散的反事实提示学习框架；它利用扩散过程，从因果模型的边缘分布和条件分布中迭代采样梯度，引导生成满足最小充分性准则的反事实。基于严格的理论推导，该方法保证反事实结果的可辨识性，同时对估计误差施加上界。我们进一步采用对比学习框架，利用所生成的反事实，从而能够精炼地提取与数据因果特征精确对齐的提示。大量实验结果表明，我们的方法在图像分类、图文检索和视觉问答等任务上表现出色，并在未见过的类别中具有尤其显著的优势。
英文摘要：
Prompt learning has garnered attention for its efficiency over traditional model training and fine-tuning. However, existing methods, constrained by inadequate theoretical foundations, encounter difficulties in achieving causally invariant prompts, ultimately falling short of capturing robust features that generalize effectively across categories. To address these challenges, we introduce the $\textit{\textbf{DiCap}}$ model, a theoretically grounded $\textbf{Di}$ffusion-based $\textbf{C}$ounterf$\textbf{a}$ctual $\textbf{p}$rompt learning framework, which leverages a diffusion process to iteratively sample gradients from the marginal and conditional distributions of the causal model, guiding the generation of counterfactuals that satisfy the minimal sufficiency criterion. Grounded in rigorous theoretical derivations, this approach guarantees the identifiability of counterfactual outcomes while imposing strict bounds on estimation errors. We further employ a contrastive learning framework that leverages the generated counterfactuals, thereby enabling the refined extraction of prompts that are precisely aligned with the causal features of the data. Extensive experimental results demonstrate that our method performs excellently across tasks such as image classification, image-text retrieval, and visual question answering, with particularly strong advantages in unseen categories.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
