LeftRefill：基于通用文本到图像扩散模型，根据左侧参考填充右侧画布
LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Model
AI总结：
本文提出 LeftRefill，将参考图与目标画布水平拼接，利用 StableDiffusion 的交叉与自注意力注入任务、视图及多参考信息，以统一框架完成参考引导修复和新视图合成。
摘要：
本文介绍了 LeftRefill，这是一种高效利用大型文本到图像（T2I）扩散模型进行参考引导图像合成的创新方法。顾名思义，LeftRefill 将参考视图和目标视图水平拼接为一个整体输入。参考图像位于左侧，目标画布位于右侧。随后，LeftRefill 根据左侧参考和特定任务指令绘制右侧目标画布。这种任务形式与上下文修复有一些相似之处，类似人类画家的动作。这一新形式能够高效学习参考与目标之间的结构和纹理对应关系，而无需其他图像编码器或适配器。我们通过 T2I 模型中的交叉注意力模块注入任务和视图信息，并通过重新排列的自注意力模块进一步展示多视图参考能力。这些设计使 LeftRefill 能够作为通用模型执行一致性生成，而无需测试时微调或模型修改。因此，LeftRefill 可被视为一个简单而统一的框架，用于解决参考引导合成问题。作为示例，我们基于预训练的 StableDiffusion，利用 LeftRefill 应对两项不同挑战：参考引导修复和新视图合成。代码和模型发布于 https://github.com/ewrfcas/LeftRefill。
英文摘要：
This paper introduces LeftRefill, an innovative approach to efficiently harness large Text-to-Image (T2I) diffusion models for reference-guided image synthesis. As the name implies, LeftRefill horizontally stitches reference and target views together as a whole input. The reference image occupies the left side, while the target canvas is positioned on the right. Then, LeftRefill paints the right-side target canvas based on the left-side reference and specific task instructions. Such a task formulation shares some similarities with contextual inpainting, akin to the actions of a human painter. This novel formulation efficiently learns both structural and textured correspondence between reference and target without other image encoders or adapters. We inject task and view information through cross-attention modules in T2I models, and further exhibit multi-view reference ability via the re-arranged self-attention modules. These enable LeftRefill to perform consistent generation as a generalized model without requiring test-time fine-tuning or model modifications. Thus, LeftRefill can be seen as a simple yet unified framework to address reference-guided synthesis. As an exemplar, we leverage LeftRefill to address two different challenges: reference-guided inpainting and novel view synthesis, based on the pre-trained StableDiffusion. Codes and models are released at https://github.com/ewrfcas/LeftRefill.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
