Qwen3-VL-Seg: 解锁基于视觉-语言接地的开放世界指代分割
Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding
AI总结：
Qwen3-VL-Seg通过视觉-语言接地框架实现开放世界指代分割，采用轻量级的框引导掩码解码器，结合多尺度空间特征注入和迭代掩码感知查询优化，实现参数高效且精确的像素级分割。
摘要：
开放世界指代分割需要将无约束的语言表达接地到精确的像素级区域。现有多模态大语言模型（MLLMs）在开放世界视觉接地方面表现强劲，但其输出仍局限于稀疏的边界框坐标，无法满足密集视觉预测需求。最近的基于MLLM的分割方法要么直接预测稀疏轮廓坐标，难以重建连续物体边界，要么依赖外部分割基础模型如Segment Anything Model（SAM），引入显著的架构和部署开销。我们提出了Qwen3-VL-Seg，一个参数高效的框架，将MLLM预测的框视为语义接地的结构先验，并将其解码为像素级指代分割。其核心是一个轻量级的框引导掩码解码器，结合多尺度空间特征注入、空间-语义查询构建、框引导的高分辨率像素融合以及迭代掩码感知查询优化，仅引入17M参数（约为基础模型的0.4%）。为了实现可扩展的开放世界训练，我们构建了SA1B-ORS，一个源自SA-1B的数据集，包含两个子集：SA1B-CoRS（类别导向样本）和SA1B-DeRS（描述性、实例特定样本）。对于评估，我们精心编排了ORS-Bench，一个人工筛选的基准，包含涵盖多样指代表达类型的在分布和不在分布子集。广泛的指代表达分割、视觉接地和ORS-Bench实验表明，Qwen3-VL-Seg在封闭集和开放世界设置中表现优异，在语言密集型指令和强不在分布泛化方面具有明显优势。在一般多模态基准上的评估进一步表明，经过分割导向适应后，模型在多模态能力方面保持广泛保留。
英文摘要：
Open-world referring segmentation requires grounding unconstrained language expressions to precise pixel-level regions. Existing multimodal large language models (MLLMs) exhibit strong open-world visual grounding, but their outputs remain limited to sparse bounding-box coordinates and are insufficient for dense visual prediction. Recent MLLM-based segmentation methods either directly predict sparse contour coordinates, struggling to reconstruct continuous object boundaries, or rely on external segmentation foundation models such as the Segment Anything Model (SAM), introducing substantial architectural and deployment overhead. We present Qwen3-VL-Seg, a parameter-efficient framework that treats the MLLM-predicted box as a semantically grounded structural prior and decodes it into pixel-level referring segmentation. At its core, a lightweight box-guided mask decoder combines multi-scale spatial feature injection, spatial-semantic query construction, box-guided high-resolution pixel fusion, and iterative mask-aware query refinement, introducing only 17M parameters (about 0.4\% of the base model). For scalable open-world training, we construct SA1B-ORS, an SA-1B-derived dataset with two subsets: SA1B-CoRS (category-oriented samples) and SA1B-DeRS (descriptive, instance-specific samples). For evaluation, we curate ORS-Bench, a manually screened benchmark with in-distribution and out-of-distribution subsets covering diverse referring expression types. Extensive experiments on referring expression segmentation, visual grounding, and ORS-Bench show that Qwen3-VL-Seg performs strongly across closed-set and open-world settings, with clear advantages on language-intensive instructions and strong out-of-distribution generalization. Evaluations on general multimodal benchmarks further show that the model broadly preserves general-purpose multimodal competence after segmentation-oriented adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
