BoxCtrl: 面向几何图像编辑的3D感知视觉提示
BoxCtrl: 3D-Aware Visual Prompting for Geometric Image Editing
AI总结：
提出BoxCtrl框架，通过投影到2D的RGB 3D边界框作为视觉提示，结合监督微调与强化学习两阶段训练，实现精确的3D几何编辑（平移、旋转、缩放等），达到最先进性能。
摘要：
随着基于指令的编辑模型和多模态大语言模型的发展，各种图像编辑任务变得可行。然而，实现精确且一致的几何图像编辑，例如在3D空间中的平移、缩放和旋转，仍然是一个重大挑战。在这项工作中，我们引入了BoxCtrl，一个3D感知的视觉提示框架。与纯文本或粗略的2D引导方法不同，我们的方法将信息丰富的RGB 3D边界框投影到2D图像上作为视觉提示。每个框的三个正交面涂有不同的RGB颜色，同时编码位置、大小和方向，提供紧凑、直观的上下文视觉示例。BoxCtrl成功的关键在于这些精心设计的边界框，它们将几何控制与外观控制解耦。这使得模型能够在潜在空间中学习相同颜色面之间的一致对应关系，从而精确理解几何意图并产生准确的编辑结果。我们引入了一个两阶段训练范式：监督微调（SFT）后接强化学习（RL）。为了解决配对数据稀缺的问题，我们构建了一个大规模合成数据集用于SFT，使模型具备基本编辑能力。为了弥合合成到真实的领域差距，我们加入了一个利用未配对真实数据的在线RL阶段。在评估几何精度和视觉保真度的奖励函数引导下，我们的SFT-RL策略显著提高了几何精度，同时保持了逼真的质量。大量实验表明，BoxCtrl在平移、旋转、缩放和复合编辑任务中均达到了最先进的性能。
英文摘要：
As instruction-based editing models and multimodal large language models advance, diverse image editing tasks have become feasible. However, achieving precise and consistent geometric image editing, such as translating, scaling, and rotating in 3D space, remains a major challenge. In this work, we introduce BoxCtrl, a 3D-aware visual prompting framework. Unlike text-only or coarse 2D-guided approaches, our method introduces informative RGB 3D bounding boxes projected onto 2D images as visual prompts. The three orthogonal faces of each box are painted with distinct RGB colors, simultaneously encoding position, size, and orientation to provide a compact, intuitive in-context visual example. The key to BoxCtrl's success lies in these well-designed bounding boxes, which decouple geometric control from appearance control. This enables the model to learn consistent correspondences between faces of the same color in the latent space, leading to a precise understanding of geometric intentions and accurate editing results. We introduce a two-stage training paradigm: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL). To address paired data scarcity, we construct a large-scale synthetic dataset for SFT, equipping the model with fundamental editing capabilities. To bridge the synthetic-to-real domain gap, we incorporate an online RL stage leveraging unpaired real-world data. Guided by a reward function evaluating geometric accuracy and visual fidelity, our SFT-RL strategy significantly enhances geometric precision while maintaining photorealistic quality. Extensive experiments demonstrate that BoxCtrl achieves state-of-the-art performance across translation, rotation, scaling, and composite editing tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
