arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.16404cs.LGcs.AIcs.CL
收藏

DynaPrompt:动态测试时提示调优

DynaPrompt: Dynamic Test-Time Prompt Tuning

AIM Lab, University of Amsterdam(阿姆斯特丹大学 AIM 实验室);Xiaohongshu Inc.(小红书公司);Tsinghua University(清华大学)

Zehao Xiao, Shilin Yan, Jack Hong, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiayi Shen, Qi Wang, Cees G. M. Snoek

AI总结:

为解决测试时提示调优忽略样本相关性及在线调优误差累积的问题,提出 DynaPrompt,通过基于预测熵和概率差的动态提示选择与追加机制,自适应优化提示缓冲区,在十四个数据集上有效提升了零样本泛化性能。

摘要:

测试时提示调优增强了视觉语言模型的零样本泛化能力,但在推理过程中往往忽略了测试样本之间的相关性。在线测试时提示调优提供了一种利用先前测试样本中信息的简单方法,但由于误差累积,存在提示崩溃的风险。为了增强测试时提示调优,我们提出了 DynaPrompt,即动态测试时提示调优的简称,它利用相关数据分布信息,同时减少误差累积。DynaPrompt 建立在线提示缓冲区之上,在调优过程中为每个测试样本自适应地选择并优化相关提示。具体而言,我们引入了一种基于两个度量指标的动态提示选择策略:预测熵和概率差。对于未见测试数据信息,我们开发了动态提示追加机制,允许缓冲区追加新提示并删除不活跃的提示。通过这样做,提示得到了优化,以利用特定测试数据上的有益信息,同时减轻误差累积。在十四个数据集上的实验证明了动态测试时提示调优的有效性。

英文摘要:

Test-time prompt tuning enhances zero-shot generalization of vision-language models but tends to ignore the relatedness among test samples during inference. Online test-time prompt tuning provides a simple way to leverage the information in previous test samples, albeit with the risk of prompt collapse due to error accumulation. To enhance test-time prompt tuning, we propose DynaPrompt, short for dynamic test-time prompt tuning, exploiting relevant data distribution information while reducing error accumulation. Built on an online prompt buffer, DynaPrompt adaptively selects and optimizes the relevant prompts for each test sample during tuning. Specifically, we introduce a dynamic prompt selection strategy based on two metrics: prediction entropy and probability difference. For unseen test data information, we develop dynamic prompt appending, which allows the buffer to append new prompts and delete the inactive ones. By doing so, the prompts are optimized to exploit beneficial information on specific test data, while alleviating error accumulation. Experiments on fourteen datasets demonstrate the effectiveness of dynamic test-time prompt tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑