PDDM：面向复杂室内场景下基于RGB-PD语义分割的伪深度扩散模型
PDDM: Pseudo Depth Diffusion Model for RGB-PD Semantic Segmentation Based in Complex Indoor Scenes
AI总结：
针对RGB-D语义分割中深度数据采集成本高、对齐难的问题，提出融合RGB与伪深度的PDDM模型及PDAM模块，在NYUv2、SUNRGB-D数据集上取得SOTA性能。
摘要：
RGB与深度模态的融合显著提升了复杂室内场景的分割精度，其中RGB-D相机采集的深度数据在这一精度提升中发挥着关键作用。然而，由于需要专用深度传感器，RGB-D数据集的采集成本远高于RGB数据集；同时，传感器位置偏差、数据缺失与噪声等问题也给深度图像与RGB图像的对齐带来了挑战。相比之下，由高精度深度估计算法生成的伪深度（Pseudo Depth，简称PD）既可以摆脱对RGB-D传感器和对齐流程的依赖，又能提供有效的深度信息，在语义分割任务中展现出巨大潜力。\n因此，为探究利用伪深度替代真实深度进行语义分割的可行性，我们设计了一套RGB-PD分割流水线以融合RGB与伪深度信息，并提出了伪深度聚合模块（Pseudo Depth Aggregation Module，简称PDAM），用于充分挖掘多幅不同伪深度图提供的信息线索。PDAM可将多幅伪深度图聚合为单一模态，使其能够轻松适配其他RGB-D分割方法。此外，预训练扩散模型可作为RGB分割任务的强效特征提取器，但基于多模态扩散的分割方法目前仍处于未被探索的阶段。为此，我们提出了伪深度扩散模型（Pseudo Depth Diffusion Model，简称PDDM），该模型采用大规模文本-图像扩散模型作为特征提取器，并通过一种简单却有效的融合策略整合伪深度信息。\n为验证伪深度及我们提出的PDDM的适用性，我们在NYUv2和SUNRGB-D数据集上开展了大量实验。实验结果表明，伪深度能够有效提升分割性能，且我们的PDDM取得了当前最优性能，在NYUv2数据集上的mIoU较其他方法高出+6.98，在SUNRGB-D数据集上的mIoU较其他方法高出+2.11。
英文摘要：
The integration of RGB and depth modalities significantly enhances the accuracy of segmenting complex indoor scenes, with depth data from RGB-D cameras playing a crucial role in this improvement. However, collecting an RGB-D dataset is more expensive than an RGB dataset due to the need for specialized depth sensors. Aligning depth and RGB images also poses challenges due to sensor positioning and issues like missing data and noise. In contrast, Pseudo Depth (PD) from high-precision depth estimation algorithms can eliminate the dependence on RGB-D sensors and alignment processes, as well as provide effective depth information and show significant potential in semantic segmentation. Therefore, to explore the practicality of utilizing pseudo depth instead of real depth for semantic segmentation, we design an RGB-PD segmentation pipeline to integrate RGB and pseudo depth and propose a Pseudo Depth Aggregation Module (PDAM) for fully exploiting the informative clues provided by the diverse pseudo depth maps. The PDAM aggregates multiple pseudo depth maps into a single modality, making it easily adaptable to other RGB-D segmentation methods. In addition, the pre-trained diffusion model serves as a strong feature extractor for RGB segmentation tasks, but multi-modal diffusion-based segmentation methods remain unexplored. Therefore, we present a Pseudo Depth Diffusion Model (PDDM) that adopts a large-scale text-image diffusion model as a feature extractor and a simple yet effective fusion strategy to integrate pseudo depth. To verify the applicability of pseudo depth and our PDDM, we perform extensive experiments on the NYUv2 and SUNRGB-D datasets. The experimental results demonstrate that pseudo depth can effectively enhance segmentation performance, and our PDDM achieves state-of-the-art performance, outperforming other methods by +6.98 mIoU on NYUv2 and +2.11 mIoU on SUNRGB-D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
