SceneCrafter：可控的多视角驾驶场景编辑
SceneCrafter: Controllable Multi-View Driving Scene Editing
AI总结：
提出SceneCrafter，基于多视角扩散模型解决驾驶场景编辑中的3D一致性与配对数据生成挑战，实现可控且逼真的多视角场景编辑。
摘要：
仿真对于开发和评估自动驾驶（AV）系统至关重要。近期文献利用新一代生成模型为全栈仿真合成高度逼真的图像。然而，纯合成的场景缺乏现实基础，难以让人信服其结果的相关性。另一方面，编辑模型利用真实驾驶日志中的源场景，能够仿真不同的交通布局、行为以及天气和时间等操作条件。虽然图像编辑是计算机视觉中的成熟课题，但它在驾驶仿真中带来了新的挑战：（1）需要跨相机的3D一致性，（2）从有前景遮挡的驾驶数据中学习“空街道”先验，（3）获取不同编辑条件下的配对图像元组，同时保持一致的布局和几何。为了解决这些挑战，我们提出了SceneCrafter，这是一个多功能的编辑器，用于对多相机捕获的驾驶场景进行逼真的3D一致操作。我们基于多视角扩散模型的最新进展，使用一个完全可控的框架，可无缝扩展到天气、时间、智能体框和高精地图等多模态条件。为了生成监督编辑模型的配对数据，我们在Prompt-to-Prompt之上提出了一个新框架，生成几何一致的合成配对数据以进行全局编辑。我们还引入了alpha混合框架来合成局部编辑数据，利用通过新型掩码训练和多视角重绘范式在空街道先验上训练的模型。SceneCrafter展示了强大的编辑能力，与现有基线相比，实现了最先进的逼真度、可控性、3D一致性和场景编辑质量。
英文摘要：
Simulation is crucial for developing and evaluating autonomous vehicle (AV) systems. Recent literature builds on a new generation of generative models to synthesize highly realistic images for full-stack simulation. However, purely synthetically generated scenes are not grounded in reality and have difficulty in inspiring confidence in the relevance of its outcomes. Editing models, on the other hand, leverage source scenes from real driving logs, and enable the simulation of different traffic layouts, behaviors, and operating conditions such as weather and time of day. While image editing is an established topic in computer vision, it presents fresh sets of challenges in driving simulation: (1) the need for cross-camera 3D consistency, (2) learning ``empty street" priors from driving data with foreground occlusions, and (3) obtaining paired image tuples of varied editing conditions while preserving consistent layout and geometry. To address these challenges, we propose SceneCrafter, a versatile editor for realistic 3D-consistent manipulation of driving scenes captured from multiple cameras. We build on recent advancements in multi-view diffusion models, using a fully controllable framework that scales seamlessly to multi-modality conditions like weather, time of day, agent boxes and high-definition maps. To generate paired data for supervising the editing model, we propose a novel framework on top of Prompt-to-Prompt to generate geometrically consistent synthetic paired data with global edits. We also introduce an alpha-blending framework to synthesize data with local edits, leveraging a model trained on empty street priors through novel masked training and multi-view repaint paradigm. SceneCrafter demonstrates powerful editing capabilities and achieves state-of-the-art realism, controllability, 3D consistency, and scene editing quality compared to existing baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
