SceneDesigner: 9自由度姿态操控的可控多物体图像生成
SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation
AI总结：
SceneDesigner通过9自由度姿态操控实现多物体图像生成，采用CNOCS图和两阶段训练策略提升可控性与质量。
摘要：
可控图像生成近年来引起了越来越多的关注，使用户能够操控视觉内容，如身份和风格。然而，同时控制多个对象的9D姿态（位置、大小和方向）仍是一个开放性挑战。尽管有最近的进展，现有方法往往在可控性和质量上有所限制，无法实现全面的多物体9D姿态控制。为了解决这些限制，我们提出了SceneDesigner，一种准确且灵活的多物体9自由度姿态操控方法。SceneDesigner将分支网络整合到预训练的基模型中，并利用一种新的表示方法CNOCS图，该方法从相机视角编码9D姿态信息。这种表示方法具有强大的几何解释特性，导致更高效和稳定的训练。为了支持训练，我们构建了一个新的数据集ObjectPose9D，该数据集聚合了来自不同来源的图像以及9D姿态注释。为进一步解决数据不平衡问题，特别是低频姿态上的性能下降，我们引入了两阶段训练策略，结合强化学习，其中第二阶段使用基于奖励的目标在重新平衡的数据上微调模型。在推理时间，我们提出了解耦物体采样技术，该技术缓解了复杂多物体场景中物体生成不足和概念混淆的问题。此外，通过整合用户特定的个性化权重，SceneDesigner能够为参考主体启用定制的姿态控制。广泛的定性和定量实验表明，SceneDesigner在可控性和质量上都显著优于现有方法。代码可在https://github.com/FudanCVL/SceneDesigner上公开获取。
英文摘要：
Controllable image generation has attracted increasing attention in recent years, enabling users to manipulate visual content such as identity and style. However, achieving simultaneous control over the 9D poses (location, size, and orientation) of multiple objects remains an open challenge. Despite recent progress, existing methods often suffer from limited controllability and degraded quality, falling short of comprehensive multi-object 9D pose control. To address these limitations, we propose SceneDesigner, a method for accurate and flexible multi-object 9-DoF pose manipulation. SceneDesigner incorporates a branched network to the pre-trained base model and leverages a new representation, CNOCS map, which encodes 9D pose information from the camera view. This representation exhibits strong geometric interpretation properties, leading to more efficient and stable training. To support training, we construct a new dataset, ObjectPose9D, which aggregates images from diverse sources along with 9D pose annotations. To further address data imbalance issues, particularly performance degradation on low-frequency poses, we introduce a two-stage training strategy with reinforcement learning, where the second stage fine-tunes the model using a reward-based objective on rebalanced data. At inference time, we propose Disentangled Object Sampling, a technique that mitigates insufficient object generation and concept confusion in complex multi-object scenes. Moreover, by integrating user-specific personalization weights, SceneDesigner enables customized pose control for reference subjects. Extensive qualitative and quantitative experiments demonstrate that SceneDesigner significantly outperforms existing approaches in both controllability and quality. Code is publicly available at https://github.com/FudanCVL/SceneDesigner.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
