arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.09447cs.CVcs.CL
收藏

基于多轮对话的像素级推理分割

Pixel-Level Reasoning Segmentation via Multi-turn Conversations

School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Singapore University of Technology and Design(新加坡科技设计大学)

Dexian Cai, Xiaocui Yang, Yongkang Liu, Daling Wang, Shi Feng, Yifei Zhang, Soujanya Poria

AI总结:

针对现有系统无法像素级推理及理解动态用户意图的问题,提出基于多轮对话的像素级推理分割任务,构建PRIST数据集并设计MIRAS框架,实验证明其在分割和推理指标上优于基线。

摘要:

现有视觉感知系统专注于单轮对话中的区域级分割,依赖复杂且显式的查询指令。这类系统无法进行像素级推理,也无法理解交互过程中动态变化的用户意图。我们的工作通过引入一项新任务——基于多轮对话的像素级推理分割(Pixel-level RS)来解决该问题,通过多轮交互跟踪演变的用户意图,以实现细粒度分割。为建立该新任务的基准,我们构建了基于多轮对话的像素级推理分割数据集(PRIST),包含来自8.3k个多轮对话场景的24k条 utterances(话语)及分割目标。基于PRIST,我们进一步提出MIRAS——多轮交互式推理分割框架,将像素级分割与稳健的多轮对话理解相集成,生成与用户意图一致的像素级解释。PRIST数据集和MIRAS框架填补了像素级推理分割的空白。PRIST数据集上的实验结果表明,我们的方法在分割指标和基于LLM的推理指标上均优于当前特定于分割的基线。代码和数据可在https://github.com/ccccai239/PixelRIST获取。

英文摘要:

Existing visual perception systems focus on region-level segmentation in single-turn dialogues, relying on complex and explicit query instructions. Such systems cannot reason at the pixel level and comprehend dynamic user intent that changes over interaction. Our work tackles this issue by introducing a novel task, Pixel-level Reasoning Segmentation (Pixel-level RS) based on multi-turn conversations, tracking evolving user intent via multi-turn interactions for fine-grained segmentation. To establish a benchmark for this novel task, we build a Pixel-level ReasonIng Segmentation Dataset Based on Multi-Turn Conversations (PRIST), comprising 24k utterances from 8.3k multi-turn conversational scenarios with segmentation targets. Building on PRIST, we further propose MIRAS, a Multi-turn Interactive ReAsoning Segmentation framework, integrates pixel-level segmentation with robust multi-turn conversation understanding, generating pixel-grounded explanations aligned with user intent. The PRIST dataset and MIRSA framework fill the gap in pixel-level reasoning segmentation. Experimental results on the PRIST dataset demonstrate that our method outperforms current segmentation-specific baselines in terms of segmentation and LLM-based reasoning metrics. The code and data are available at: https://github.com/ccccai239/PixelRIST.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑