arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.10316cs.CVcs.GR
收藏

Analogist:基于图像扩散模型的开箱即用视觉上下文学习

Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion Model

City University of Hong Kong(香港城市大学);State Key Lab for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Tianjin University(天津大学)

Zheng Gu, Shiyuan Yang, Jing Liao, Jing Huo, Yang Gao

AI总结:

针对现有视觉ICL泛化性差、依赖文本提示效率低等问题,提出开箱即用的Analogist方法,结合视觉与文本提示技术,无需微调即可高效完成多类视觉任务,性能优于现有方法。

摘要:

视觉上下文学习(ICL)凭借类比推理能力,仅需少量示例对即可完成各类任务,已成为颇具前景的研究领域。然而,基于训练的视觉ICL在泛化至未见过的任务方面存在局限性,且需要收集多样化的任务数据集。另一方面,现有基于推理的视觉ICL方法仅依赖文本提示,无法从给定示例中捕捉细粒度上下文信息,且将图像转换为文本提示的过程较为耗时。为解决这些挑战,我们提出Analogist——一种新型基于推理的视觉ICL方法,它利用经图像修复预训练的文生图扩散模型,同时结合视觉与文本提示技术。在视觉提示方面,我们提出自注意力克隆(SAC)方法,用于引导图像示例间细粒度结构层面的类比。在文本提示方面,我们利用GPT-4V的视觉推理能力高效生成文本提示,并引入交叉注意力掩码(CAM)操作,以提升文本提示引导下语义层面类比的准确性。我们的方法是开箱即用的,无需微调或优化;同时具备通用性与灵活性,能够以上下文学习的方式执行广泛的视觉任务。大量实验从定性和定量两方面证明了我们的方法优于现有方法。

英文摘要:

Visual In-Context Learning (ICL) has emerged as a promising research area due to its capability to accomplish various tasks with limited example pairs through analogical reasoning. However, training-based visual ICL has limitations in its ability to generalize to unseen tasks and requires the collection of a diverse task dataset. On the other hand, existing methods in the inference-based visual ICL category solely rely on textual prompts, which fail to capture fine-grained contextual information from given examples and can be time-consuming when converting from images to text prompts. To address these challenges, we propose Analogist, a novel inference-based visual ICL approach that exploits both visual and textual prompting techniques using a text-to-image diffusion model pretrained for image inpainting. For visual prompting, we propose a self-attention cloning (SAC) method to guide the fine-grained structural-level analogy between image examples. For textual prompting, we leverage GPT-4V's visual reasoning capability to efficiently generate text prompts and introduce a cross-attention masking (CAM) operation to enhance the accuracy of semantic-level analogy guided by text prompts. Our method is out-of-the-box and does not require fine-tuning or optimization. It is also generic and flexible, enabling a wide range of visual tasks to be performed in an in-context manner. Extensive experiments demonstrate the superiority of our method over existing approaches, both qualitatively and quantitatively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑