SCPainter：一个用于真实3D资产插入和新视角合成的统一框架
SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesis
AI总结：
SCPainter通过整合3D高斯点和点云与扩散生成，实现真实3D资产插入与新视角合成，提升自动驾驶训练数据的多样性和真实性。
摘要：
3D资产插入和新视角合成（NVS）是自动驾驶模拟的关键组成部分，有助于增强训练数据的多样性。通过更丰富、覆盖广泛情境的训练数据，包括长尾驾驶场景，自动驾驶模型可以变得更加鲁棒和安全。这促使了一个统一的模拟框架，能够共同处理真实插入的3D资产和NVS。最近的3D资产重建方法能够从视频中重建动态角色，支持其重新插入到模拟驾驶场景中。尽管整体结构和外观可以准确，但仍难以通过光照或阴影捕捉3D资产的真实感，特别是在插入到场景中时。同时，最近的NVS方法进展在合成超出原始记录轨迹的视角方面表现出有希望的结果。然而，现有方法大多将资产插入和NVS能力孤立处理。为了与场景的其余部分进行交互，并使训练数据的多样化创建成为可能，真实3D资产插入应与NVS相结合。为此，我们提出了SCPainter（街道汽车画家），一个统一的框架，整合了3D高斯点（GS）汽车资产表示和3D场景点云，与扩散生成相结合，以共同实现真实3D资产插入和NVS。3D GS资产和3D场景点云被同时投影到新视角中，这些投影用于条件化扩散模型以生成高质量图像。在Waymo开放数据集上的评估证明了我们的框架能够实现3D资产插入和NVS的能力，促进了多样且真实的驾驶数据创建。
英文摘要：
3D Asset insertion and novel view synthesis (NVS) are key components for autonomous driving simulation, enhancing the diversity of training data. With better training data that is diverse and covers a wide range of situations, including long-tailed driving scenarios, autonomous driving models can become more robust and safer. This motivates a unified simulation framework that can jointly handle realistic integration of inserted 3D assets and NVS. Recent 3D asset reconstruction methods enable reconstruction of dynamic actors from video, supporting their re-insertion into simulated driving scenes. While the overall structure and appearance can be accurate, it still struggles to capture the realism of 3D assets through lighting or shadows, particularly when inserted into scenes. In parallel, recent advances in NVS methods have demonstrated promising results in synthesizing viewpoints beyond the originally recorded trajectories. However, existing approaches largely treat asset insertion and NVS capabilities in isolation. To allow for interaction with the rest of the scene and to enable more diverse creation of new scenarios for training, realistic 3D asset insertion should be combined with NVS. To address this, we present SCPainter (Street Car Painter), a unified framework which integrates 3D Gaussian Splat (GS) car asset representations and 3D scene point clouds with diffusion-based generation to jointly enable realistic 3D asset insertion and NVS. The 3D GS assets and 3D scene point clouds are projected together into novel views, and these projections are used to condition a diffusion model to generate high quality images. Evaluation on the Waymo Open Dataset demonstrate the capability of our framework to enable 3D asset insertion and NVS, facilitating the creation of diverse and realistic driving data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
