arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.08841cs.AIcs.CV
收藏

探索视觉上下文学习的任务级最优提示

Exploring Task-Level Optimal Prompts for Visual In-Context Learning

Tianjin University(天津大学)

Yan Zhu, Huan Ma, Changqing Zhang

AI总结:

针对视觉上下文学习中样本级最优提示搜索成本过高的问题,本文发现多数样本共享最优提示,提出任务级提示及两种高效搜索策略,以极低实现成本达到最优VICL性能。

摘要:

近年来,随着视觉基础模型(Vision Foundation Models, VFMs)的发展,视觉上下文学习(Visual In-Context Learning, VICL)在大多数场景下已成为比修改模型更优的选择。与重新训练或微调模型不同,VICL无需修改模型的权重或架构,仅需一个带有示例的提示来指导VFM解决任务。目前,为每个测试样本寻找最优提示的巨大计算成本阻碍了VICL的部署,因为确定使用哪些示例构建提示的成本极高。然而,本文中我们发现了一个反直觉的现象:大多数测试样本实际上在相同的提示下能达到最优性能,搜索样本级提示只会耗费更多时间,最终得到的提示却完全相同。因此,我们提出任务级提示方法以降低推理阶段的提示搜索成本,并引入两种省时且有效的任务级提示搜索策略。大量实验结果表明,我们提出的方法能够找到接近最优的提示,并以现有工作从未实现的最低成本达到最佳的VICL性能。

英文摘要:

With the development of Vision Foundation Models (VFMs) in recent years, Visual In-Context Learning (VICL) has become a better choice compared to modifying models in most scenarios. Different from retraining or fine-tuning model, VICL does not require modifications to the model's weights or architecture, and only needs a prompt with demonstrations to teach VFM how to solve tasks. Currently, significant computational cost for finding optimal prompts for every test sample hinders the deployment of VICL, as determining which demonstrations to use for constructing prompts is very costly. In this paper, however, we find a counterintuitive phenomenon that most test samples actually achieve optimal performance under the same prompts, and searching for sample-level prompts only costs more time but results in completely identical prompts. Therefore, we propose task-level prompting to reduce the cost of searching for prompts during the inference stage and introduce two time-saving yet effective task-level prompt search strategies. Extensive experimental results show that our proposed method can identify near-optimal prompts and reach the best VICL performance with a minimal cost that prior work has never achieved.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑