arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.04748cs.CV
收藏

探索提升视觉上下文学习的有效因素

Exploring Effective Factors for Improving Visual In-Context Learning

Baidu VIS(百度视觉团队);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)

Yanpeng Sun, Qiang Chen, Xiaofan Li, Jian Wang, Jingdong Wang, Zechao Li

AI总结:

本文揭示了提示选择与提示融合是影响视觉上下文学习性能的关键因素,提出prompt-SelF框架,通过像素级检索选取提示并融合不同位置知识,首次在1-shot分割任务中超越基于OSLSM的元学习方法。

摘要:

上下文学习(ICL)旨在通过少量示例(即提示)理解新任务,并在不微调模型的情况下预测新输入。尽管该技术在自然语言处理(NLP)领域已被广泛研究,但在计算机视觉领域仍是一个相对新兴的研究方向。为揭示影响视觉上下文学习性能的因素,本文表明提示选择与提示融合是两大关键因素,它们对视觉上下文学习的推理性能具有直接影响。提示选择是指寻找最合适的提示或示例以辅助模型理解新任务的过程。这一环节至关重要,因为向模型提供相关提示能够帮助其更高效、有效地学习。提示融合则涉及整合大规模视觉模型中不同位置的知识。通过这种方式,模型可以充分利用存储在其不同部分的多样化知识,从而提升在新任务上的表现。基于上述发现,我们提出了一个用于视觉上下文学习的简单框架 prompt-SelF。具体而言,我们首先采用像素级检索方法选取合适的提示,随后利用不同的提示融合方法激活存储于大规模模型中的所有知识,最后对不同提示融合方法得到的预测结果进行集成,以获得最终预测。我们在单目标分割与检测任务上开展了大量实验,证明了 prompt-SelF 的有效性。值得注意的是,prompt-SelF 首次在 1-shot 分割任务中超越了基于 OSLSM 的元学习方法。这表明了视觉上下文学习的巨大潜力。源代码和模型将在 https://github.com/syp2ysy/prompt-SelF 公开。

英文摘要:

The In-Context Learning (ICL) is to understand a new task via a few demonstrations (aka. prompt) and predict new inputs without tuning the models. While it has been widely studied in NLP, it is still a relatively new area of research in computer vision. To reveal the factors influencing the performance of visual in-context learning, this paper shows that prompt selection and prompt fusion are two major factors that have a direct impact on the inference performance of visual context learning. Prompt selection is the process of identifying the most appropriate prompt or example to help the model understand new tasks. This is important because providing the model with relevant prompts can help it learn more effectively and efficiently. Prompt fusion involves combining knowledge from different positions within the large-scale visual model. By doing this, the model can leverage the diverse knowledge stored in different parts of the model to improve its performance on new tasks. Based these findings, we propose a simple framework prompt-SelF for visual in-context learning. Specifically, we first use the pixel-level retrieval method to select a suitable prompt, and then use different prompt fusion methods to activate all the knowledge stored in the large-scale model, and finally ensemble the prediction results obtained from different prompt fusion methods to obtain the final prediction results. And we conduct extensive experiments on single-object segmentation and detection tasks to demonstrate the effectiveness of prompt-SelF. Remarkably, the prompt-SelF has outperformed OSLSM based meta-learning in 1-shot segmentation for the first time. This indicated the great potential of visual in-context learning. The source code and models will be available at https://github.com/syp2ysy/prompt-SelF.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑