In-Context Brush：基于上下文感知潜在空间操作的零样本定制主体插入
In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation
AI总结：
提出零样本框架In-Context Brush，通过上下文学习和双层潜在空间操作，实现无需微调的高保真定制主体插入，优于现有方法。
摘要：
扩散模型的最新进展增强了多模态引导的视觉生成，使得定制主体插入成为可能，即根据文本提示，将用户指定的对象无缝地“刷入”给定图像中。然而，现有方法在插入定制主体时往往难以实现高保真度，并通过文本提示使结果与用户意图对齐。在这项工作中，我们提出了“In-Context Brush”，一种零样本的定制主体插入框架，通过将任务重新表述为上下文学习的范式来实现。不失一般性，我们将对象图像和文本提示作为跨模态演示，将带有掩码区域的目标图像作为查询。目标是修复目标图像，使主体与文本提示对齐，而无需模型微调。基于预训练的MMDiT修复网络，我们通过双层潜在空间操作进行测试时增强：注意力头内的“潜在特征移位”，动态调整注意力输出以反映所需的主体语义；以及跨不同注意力头的“注意力重新加权”，通过差异化注意力优先级增强提示可控性。大量实验和应用表明，与现有最先进方法相比，我们的方法在身份保持、文本对齐和图像质量方面表现出优越性，且无需专门的训练或额外的数据收集。
英文摘要：
Recent advances in diffusion models have enhanced multimodal-guided visual generation, enabling customized subject insertion that seamlessly "brushes" user-specified objects into a given image guided by textual prompts. However, existing methods often struggle to insert customized subjects with high fidelity and align results with the user's intent through textual prompts. In this work, we propose "In-Context Brush", a zero-shot framework for customized subject insertion by reformulating the task within the paradigm of in-context learning. Without loss of generality, we formulate the object image and the textual prompts as cross-modal demonstrations, and the target image with the masked region as the query. The goal is to inpaint the target image with the subject aligning textual prompts without model tuning. Building upon a pretrained MMDiT-based inpainting network, we perform test-time enhancement via dual-level latent space manipulation: intra-head "latent feature shifting" within each attention head that dynamically shifts attention outputs to reflect the desired subject semantics and inter-head "attention reweighting" across different heads that amplifies prompt controllability through differential attention prioritization. Extensive experiments and applications demonstrate that our approach achieves superior identity preservation, text alignment, and image quality compared to existing state-of-the-art methods, without requiring dedicated training or additional data collection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
