OpenSeg-R：通过逐步视觉推理改进开放词汇分割
OpenSeg-R: Improving Open-Vocabulary Segmentation via Step-by-Step Visual Reasoning
AI总结：
针对现有开放词汇分割方法缺乏显式推理、难以区分相似类别的问题，提出OpenSeg-R框架，借助大型多模态模型进行分层逐步视觉推理生成提示，在多个基准的语义与全景分割任务上取得性能提升，同时增强了可解释性。
摘要：
开放词汇分割（Open-Vocabulary Segmentation, OVS）因具备将分割能力泛化至预定义类别之外的特性而受到越来越多的关注。然而，现有方法通常通过简单的前向推理预测分割掩码，缺乏显式推理过程与可解释性。这导致OVS模型在开放世界场景下，由于缺乏上下文理解与具有区分度的视觉线索，难以区分相似类别。为解决这一局限，我们提出一种面向开放词汇分割的逐步视觉推理框架，命名为OpenSeg-R。该框架利用大型多模态模型（Large Multimodal Models, LMMs）在分割前执行分层视觉推理。具体而言，我们为每张图像生成通用推理与图像特定推理，形成结构化三元组，以从粗到细的方式解释物体的视觉成因。基于这些推理步骤，我们可以构建详细的描述提示词，并将其输入分割器以生成更准确的分割掩码。据我们所知，OpenSeg-R是首个将显式逐步视觉推理引入OVS的框架。实验结果表明，在五个基准数据集的开放词汇语义分割任务上，OpenSeg-R的性能显著优于当前最优方法。此外，它在开放词汇全景分割的所有指标上均取得了一致的性能提升。定性结果进一步凸显了我们的推理引导框架在提升分割精度与可解释性两方面的有效性。我们的代码已公开于https://github.com/Hanzy1996/OpenSeg-R。
英文摘要：
Open-Vocabulary Segmentation (OVS) has drawn increasing attention for its capacity to generalize segmentation beyond predefined categories. However, existing methods typically predict segmentation masks with simple forward inference, lacking explicit reasoning and interpretability. This makes it challenging for OVS model to distinguish similar categories in open-world settings due to the lack of contextual understanding and discriminative visual cues. To address this limitation, we propose a step-by-step visual reasoning framework for open-vocabulary segmentation, named OpenSeg-R. The proposed OpenSeg-R leverages Large Multimodal Models (LMMs) to perform hierarchical visual reasoning before segmentation. Specifically, we generate both generic and image-specific reasoning for each image, forming structured triplets that explain the visual reason for objects in a coarse-to-fine manner. Based on these reasoning steps, we can compose detailed description prompts, and feed them to the segmentor to produce more accurate segmentation masks. To the best of our knowledge, OpenSeg-R is the first framework to introduce explicit step-by-step visual reasoning into OVS. Experimental results demonstrate that OpenSeg-R significantly outperforms state-of-the-art methods on open-vocabulary semantic segmentation across five benchmark datasets. Moreover, it achieves consistent gains across all metrics on open-vocabulary panoptic segmentation. Qualitative results further highlight the effectiveness of our reasoning-guided framework in improving both segmentation precision and interpretability. Our code is publicly available at https://github.com/Hanzy1996/OpenSeg-R.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
