通过文本可控的图像到3D进行前馈编辑
Feedforward 3D Editing via Text-Steerable Image-to-3D
AI总结：
Steer3D通过文本可控的图像到3D生成方法，实现高效且准确的3D资产编辑，速度提升显著。
摘要：
最近在图像到3D领域的进展为设计、AR/VR和机器人领域打开了巨大的可能性。然而，要将AI生成的3D资产用于实际应用，一个关键要求是能够轻松编辑它们。我们提出了一种前馈方法Steer3D，用于向图像到3D模型添加文本可控性，这使能够通过语言编辑生成的3D资产。我们的方法受到ControlNet的启发，我们将其适应到图像到3D生成中，以在前向传递中直接实现文本控制。我们构建了一个可扩展的数据引擎用于自动数据生成，并开发了基于流匹配训练和直接偏好优化（DPO）的两阶段训练配方。与竞争方法相比，Steer3D更忠实遵循语言指令，并且在与原始3D资产保持一致方面表现更好，同时速度提高了2.4倍至28.5倍。Steer3D证明了通过10万数据将新的模态（文本）添加到预训练的图像到3D生成模型中是可行的。项目网站：https://glab-caltech.github.io/steer3d/
英文摘要：
Recent progress in image-to-3D has opened up immense possibilities for design, AR/VR, and robotics. However, to use AI-generated 3D assets in real applications, a critical requirement is the capability to edit them easily. We present a feedforward method, Steer3D, to add text steerability to image-to-3D models, which enables editing of generated 3D assets with language. Our approach is inspired by ControlNet, which we adapt to image-to-3D generation to enable text steering directly in a forward pass. We build a scalable data engine for automatic data generation, and develop a two-stage training recipe based on flow-matching training and Direct Preference Optimization (DPO). Compared to competing methods, Steer3D more faithfully follows the language instruction and maintains better consistency with the original 3D asset, while being 2.4x to 28.5x faster. Steer3D demonstrates that it is possible to add a new modality (text) to steer the generation of pretrained image-to-3D generative models with 100k data. Project website: https://glab-caltech.github.io/steer3d/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
