图像在图像中说话：用于情境视觉学习的通才画家
Images Speak in Images: A Generalist Painter for In-Context Visual Learning
摘要：
上下文学习作为自然语言处理中的一种新范式，它允许模型在只有少量提示和示例的情况下快速适应各种任务。但在计算机视觉中，上下文学习的难点在于任务在输出表征上的差异很大，因此如何定义视觉模型能够理解的通用任务提示并将其转换为域外任务还不清楚。在这项工作中，我们提出了画家，一个通才模型，它解决了这些障碍与“图像”为中心的解决方案，即，重新定义输出的核心视觉任务的图像，并指定任务提示也图像。有了这个想法，我们的训练过程非常简单，它对输入和输出图像对的拼接执行标准的掩码图像建模。这使得模型能够执行以可见图像补丁为条件的任务。因此，在推理过程中，我们可以采用来自同一任务的一对输入和输出图像作为输入条件，以指示执行哪个任务。没有花里胡哨，我们的多面手画家可以实现竞争力的表现相比，完善的特定任务的模型，在七个代表性的视觉任务，从高层次的视觉理解到低层次的图像处理。此外，Painter在几个具有挑战性的任务上明显优于最近的通才模型。
英文摘要：
In-context learning, as a new paradigm in NLP, allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. But in computer vision, the difficulties for in-context learning lie in that tasks vary significantly in the output representations, thus it is unclear how to define the general-purpose task prompts that the vision model can understand and transfer to out-of-domain tasks. In this work, we present Painter, a generalist model which addresses these obstacles with an "image"-centric solution, that is, to redefine the output of core vision tasks as images, and specify task prompts as also images. With this idea, our training process is extremely simple, which performs standard masked image modeling on the stitch of input and output image pairs. This makes the model capable of performing tasks conditioned on visible image patches. Thus, during inference, we can adopt a pair of input and output images from the same task as the input condition, to indicate which task to perform. Without bells and whistles, our generalist Painter can achieve competitive performance compared to well-established task-specific models, on seven representative vision tasks ranging from high-level visual understanding to low-level image processing. In addition, Painter significantly outperforms recent generalist models on several challenging tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
