近年来,随着视觉基础模型(Vision Foundation Models, VFMs)的发展,视觉上下文学习(Visual In-Context Learning, VICL)在大多数场景下已成为比修改模型更优的选择。与重新训练或微调模型不同,VICL无需修改模型的权重或架构,仅需一个带有示例的提示来指导VFM解决任务。目前,为每个测试样本寻找最优提示的巨大计算成本阻碍了VICL的部署,因为确定使用哪些示例构建提示的成本极高。然而,本文中我们发现了一个反直觉的现象:大多数测试样本实际上在相同的提示下能达到最优性能,搜索样本级提示只会耗费更多时间,最终得到的提示却完全相同。因此,我们提出任务级提示方法以降低推理阶段的提示搜索成本,并引入两种省时且有效的任务级提示搜索策略。大量实验结果表明,我们提出的方法能够找到接近最优的提示,并以现有工作从未实现的最低成本达到最佳的VICL性能。
英文摘要:
With the development of Vision Foundation Models (VFMs) in recent years, Visual In-Context Learning (VICL) has become a better choice compared to modifying models in most scenarios. Different from retraining or fine-tuning model, VICL does not require modifications to the model's weights or architecture, and only needs a prompt with demonstrations to teach VFM how to solve tasks. Currently, significant computational cost for finding optimal prompts for every test sample hinders the deployment of VICL, as determining which demonstrations to use for constructing prompts is very costly. In this paper, however, we find a counterintuitive phenomenon that most test samples actually achieve optimal performance under the same prompts, and searching for sample-level prompts only costs more time but results in completely identical prompts. Therefore, we propose task-level prompting to reduce the cost of searching for prompts during the inference stage and introduce two time-saving yet effective task-level prompt search strategies. Extensive experimental results show that our proposed method can identify near-optimal prompts and reach the best VICL performance with a minimal cost that prior work has never achieved.
Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning
Fudan University(复旦大学);Alibaba Group Holding Limited(阿里巴巴集团控股有限公司);Future Living Lab of Alibaba(阿里巴巴未来生活实验室);University of Southern California(南加州大学);Shanghai Innovation Institute(上海创新研究院)
Review of Large Vision Models and Visual Prompt Engineering
Northwestern Polytechnical University(西北工业大学);The University of Georgia(佐治亚大学);City University of Hong Kong(香港城市大学);Shaanxi Normal University(陕西师范大学);University of Electronic Science and Technology of China(电子科技大学);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
DualCap: Enhancing Lightweight Image Captioning via Dual Retrieval with Similar Scenes Visual Prompts
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation
MMLab, CUHK(香港中文大学多媒体实验室);GVC Lab, Great Bay University(大湾区大学GVC实验室);ARC Lab, Tencent PCG(腾讯平台与内容事业群ARC实验室);Tencent AI Lab(腾讯AI实验室);SHIAE, CUHK(香港中文大学SHIAE实验室)
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别新实验室)
Improving Visual Object Tracking through Visual Prompting
Department of Computer Science, National Yang Ming Chiao Tung University(国立阳明交通大学计算机科学系);Research Center for Information Technology Innovation, Academia Sinica, Taiwan(学术院资讯科技创新研究中心);Microsoft, Seattle, Washington, United States(微软公司)
Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language
University of Auckland(奥克兰大学);Indian Institute of Science(印度科学学院)
GitHub Copilot是一个人工智能模型,用于从自然语言问题描述自动生成源代码。自2022年6月以来,Copilot已正式免费提供给所有学生作为Visual Studio Code等开发环境的插件。之前探索OpenAI Codex(Copilot的底层模型)的工作表明,它在典型的CS1问题上表现良好,因此人们担心它会对入门编程课程的教学产生影响。然而,关于Copilot不能很好地执行的问题类型,或者关于学生在解决错误时可能与Copilot进行的自然语言交互,我们知之甚少。我们通过评估Copilot在166个编程问题的公开数据集上的性能来探索这些问题。我们发现,它成功地解决了大约一半的这些问题的第一次尝试,它解决了60%的剩余问题,只使用自然语言的问题描述的变化。我们认为,这种类型的提示工程,我们相信将成为一个标准的人类和Copilot之间的互动时,它最初失败,是一个潜在的有用的学习活动,促进计算思维技能,并有可能改变代码编写技能的发展的性质。