arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.18795cs.CVcs.AI
收藏

Perceptio:通过空间标记生成增强视觉语言模型

Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation

Amazon(亚马逊)

Yuchen Li, Amanmeet Garg, Shalini Chaudhuri, Rui Zhao, Garin Kessler

AI总结:

Perceptio通过生成2D和3D空间标记提升视觉语言模型的空间推理能力,结合语义分割和深度标记实现更精确的空间理解,提升多个基准测试性能。

摘要:

Perceptio通过生成2D和3D空间标记提升视觉语言模型的空间推理能力,结合语义分割和深度标记实现更精确的空间理解,提升多个基准测试性能。

英文摘要:

Large Vision Language Models (LVLMs) excel at semantic understanding but struggle with fine grained spatial grounding, as the model must implicitly infer complex geometry without ever producing a spatial interpretation. We present Perceptio, a perception enhanced LVLM with 2D and 3D spatial reasoning abilities, enabled via explicit semantic segmentation tokens and depth tokens generated directly within the autoregressive sequence. Concretely, we (i) distill a VQVAE depth codebook from a strong monocular teacher to tokenize dense depth into compact sequences, and (ii) integrate SAM2 based semantic segmentation tokens and VQ-VAE depth tokens inside the LLM so the model first emits spatial tokens and then answers. To stabilize depth token generation, we introduce novel composite depth-token objectives (marker, token, and count losses) and a soft-merging technique for differentiable reconstruction. We adopt a multi-task co-training strategy across diverse datasets, letting the model learn perception tokens to tackle multiple downstream tasks. Building on InternVL, Perceptio achieves state-of-the-art performance across benchmarks: improving referring expression segmentation by +0.8/+1.4/+1.1 cIoU on RefCOCO/+/g HardBLINK spatial understanding accuracy by 10.3%, and MMBench accuracy by 1.0%, demonstrating that explicit spatial chain-of-thought materially strengthens spatial grounding in LVLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑