arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.04995cs.CVcs.AI
收藏

IPDN:用于3D指代表达分割的图像增强提示解码网络

IPDN: Image-enhanced Prompt Decoding Network for 3D Referring Expression Segmentation

Xiamen University(厦门大学);National University of Singapore(新加坡国立大学)

Qi Chen, Changli Wu, Jiayi Ji, Yiwei Ma, Danni Yang, Xiaoshuai Sun

AI总结:

针对3D指代表达分割中的特征与意图歧义,提出IPDN,通过多视图语义嵌入和提示感知解码器增强3D场景理解,并在3D-RES、3D-GRES上分别提升mIoU 1.9和4.2个点。

摘要:

3D指代表达分割(3D-RES)旨在根据给定表达对点云场景进行分割。然而,现有3D-RES方法面临两大挑战:特征歧义与意图歧义。特征歧义源于点云采集过程中因光照和视点等限制而导致的信息丢失或失真。意图歧义则指模型在解码过程中对所有查询一视同仁,缺乏自上而下的特定任务引导。本文提出一种图像增强提示解码网络(Image enhanced Prompt Decoding Network,IPDN),利用多视图图像和任务驱动信息增强模型的推理能力。为解决特征歧义,我们提出多视图语义嵌入(Multi-view Semantic Embedding,MSE)模块,将多视图2D图像信息注入3D场景,并补偿潜在的空间信息丢失。为应对意图歧义,我们设计了提示感知解码器(Prompt-Aware Decoder,PAD),通过从表达与视觉特征的交互中获取任务驱动信号来引导解码过程。大量实验表明,IPDN在3D-RES和3D-GRES任务上的mIoU指标分别优于当前最优方法1.9和4.2个点。

英文摘要:

3D Referring Expression Segmentation (3D-RES) aims to segment point cloud scenes based on a given expression. However, existing 3D-RES approaches face two major challenges: feature ambiguity and intent ambiguity. Feature ambiguity arises from information loss or distortion during point cloud acquisition due to limitations such as lighting and viewpoint. Intent ambiguity refers to the model's equal treatment of all queries during the decoding process, lacking top-down task-specific guidance. In this paper, we introduce an Image enhanced Prompt Decoding Network (IPDN), which leverages multi-view images and task-driven information to enhance the model's reasoning capabilities. To address feature ambiguity, we propose the Multi-view Semantic Embedding (MSE) module, which injects multi-view 2D image information into the 3D scene and compensates for potential spatial information loss. To tackle intent ambiguity, we designed a Prompt-Aware Decoder (PAD) that guides the decoding process by deriving task-driven signals from the interaction between the expression and visual features. Comprehensive experiments demonstrate that IPDN outperforms the state-ofthe-art by 1.9 and 4.2 points in mIoU metrics on the 3D-RES and 3D-GRES tasks, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑