TecoPrompt：面向视觉-语言模型的时间保守提示学习
TecoPrompt: Temporal-Conservative Prompt Learning for Vision-Language Models
浏览论文内容
中文总结 AI 辅助
针对少样本提示学习易受标签噪声干扰的问题，提出时间保守的闭环框架TecoPrompt，利用最优传输伪标签和轨迹稳定性验证，在多个噪声数据集上显著提升性能。
中文摘要 AI 辅助
提示学习通过调整少量上下文标记来适配视觉-语言模型（如CLIP）。然而，在少样本监督下，即使是中等程度的标签噪声也可能破坏提示优化。为解决这一问题，我们提出了TecoPrompt，一种闭环鲁棒提示学习框架，从时间角度重新审视最优传输（OT）伪标签。TecoPrompt在CLIP语义空间中采用熵正则化OT方案，以获得全局一致的标签候选。它通过检查轨迹稳定性来验证这些候选的可靠性：仅当OT候选在K个epoch的时间稳定性窗口内保持不变，并通过基于指数移动平均（EMA）的置信度门控时，噪声标签才会被重写。这种方法有助于减少确认偏差。重写后的标签随后通过一个三组目标函数集成回提示训练中，该目标函数包含与干净、中等和噪声子集对齐的三个损失函数。在七个数据集上进行的实验，包括合成对称和非对称噪声以及Food101N，展示了显著的性能提升。例如，在OxfordPets数据集上，在50%非对称噪声下，TecoPrompt的准确率从0.775提升至0.843。
英文摘要
Prompt learning adapts vision-language models, such as CLIP, by adjusting a small set of context tokens. However, under few-shot supervision, even moderate label noise can disrupt prompt optimization. To address this issue, we propose TecoPrompt, a closed-loop robust prompt-learning framework that revisits optimal transport (OT) pseudo-labeling from a temporal perspective. TecoPrompt employs an entropic OT plan in the CLIP semantic space to obtain globally consistent label candidates. It verifies the reliability of these candidates by examining trajectory stability: a noisy label is only rewritten if the OT candidate remains unchanged within a K-epoch temporal stability window and passes a confidence gate based on Exponential Moving Average (EMA). This approach helps reduce confirmation bias. The rewritten labels are then integrated back into prompt training using a tri-group objective that includes three loss functions aligned with clean, mid, and noisy subsets. Experiments on seven datasets with synthetic symmetric and asymmetric noise, as well as Food101N, demonstrate significant performance improvements. For example, on the OxfordPets dataset, with 50% asymmetric noise, TecoPrompt achieves an accuracy of 0.843, up from 0.775.
发表机构
- Soochow University（苏州大学）
- NVIDIA（英伟达）
- Heriot-Watt University Malaysia（赫瑞-瓦特大学马来西亚校区）
- Fraunhofer FIT（弗劳恩霍夫应用信息技术研究所）
机构由 AI 辅助整理，请以论文原文为准。
