面向多模态反事实生成的参考引导扩散修复
Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation
AI总结：
本研究提出MObI和AnydoorMed两种参考引导扩散修复方法，分别用于自动驾驶多模态目标插入与乳腺钼靶异常修复，验证了自然图像修复基础模型可适配多模态，助力构建高真实感可控的多模态反事实场景。
摘要：
自动驾驶和医学图像分析等安全关键应用需要大量多模态数据以开展严格测试。由于采集真实世界数据的成本高且复杂度大，合成数据方法正日益受到重视，但这类方法需要具备高度的真实感与可控性才能发挥作用。本研究针对自动驾驶和医学图像分析领域，分别提出了两种全新的合成数据生成方法，即MObI和AnydoorMed。MObI是首个多模态目标修复（Multimodal Object Inpainting）框架，它利用扩散模型生成跨感知模态的真实可控目标修复结果，已在相机和激光雷达场景中同时得到验证。仅需一张参考RGB图像，MObI就能在边界框的引导下，将目标无缝插入现有多模态场景的指定3D位置，同时保持语义一致性与多模态连贯性。与仅依赖编辑掩码的传统修复方法不同，该方法采用3D边界框条件约束，以确保精准的空间定位和真实的缩放比例。AnydoorMed将这一范式拓展至医学成像领域，聚焦于乳腺钼靶扫描的参考引导修复任务。它利用基于扩散的模型修复异常区域，在细节保留方面表现出色，既维持了参考异常的结构完整性，又能使其与周围组织实现语义层面的融合。这些方法共同证明，自然图像领域的参考引导修复基础模型可轻松适配多种感知模态，为下一代能够构建高度真实、可控且多模态的反事实场景的系统铺平了道路。
英文摘要：
Safety-critical applications, such as autonomous driving and medical image analysis, require extensive multimodal data for rigorous testing. Synthetic data methods are gaining prominence due to the cost and complexity of gathering real-world data, but they demand a high degree of realism and controllability to be useful. This work introduces two novel methods for synthetic data generation in autonomous driving and medical image analysis, namely MObI and AnydoorMed, respectively. MObI is a first-of-its-kind framework for Multimodal Object Inpainting that leverages a diffusion model to produce realistic and controllable object inpaintings across perceptual modalities, demonstrated simultaneously for camera and lidar. Given a single reference RGB image, MObI enables seamless object insertion into existing multimodal scenes at a specified 3D location, guided by a bounding box, while maintaining semantic consistency and multimodal coherence. Unlike traditional inpainting methods that rely solely on edit masks, this approach uses 3D bounding box conditioning to ensure accurate spatial positioning and realistic scaling. AnydoorMed extends this paradigm to the medical imaging domain, focusing on reference-guided inpainting for mammography scans. It leverages a diffusion-based model to inpaint anomalies with impressive detail preservation, maintaining the reference anomaly's structural integrity while semantically blending it with the surrounding tissue. Together, these methods demonstrate that foundation models for reference-guided inpainting in natural images can be readily adapted to diverse perceptual modalities, paving the way for the next generation of systems capable of constructing highly realistic, controllable and multimodal counterfactual scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
