arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.02870cs.CV
收藏

令牌扭曲帮助多模态大语言模型从近处视角观察

Token Warping Helps MLLMs Look from Nearby Viewpoints

KAIST(韩国科学技术院)

Phillip Y. Lee, Chanho Park, Mingue Park, Seungwoo Yoo, Juil Koo, Minhyuk Sung

AI总结:

本文研究令牌扭曲如何帮助多模态大语言模型从近处视角理解场景,发现反向令牌扭曲在视角变化中更稳定且能更好保持语义一致性。

摘要:

本文研究令牌扭曲如何帮助多模态大语言模型从近处视角理解场景,发现反向令牌扭曲在视角变化中更稳定且能更好保持语义一致性。

英文摘要:

Can warping tokens, rather than pixels, help multimodal large language models (MLLMs) understand how a scene appears from a nearby viewpoint? While MLLMs perform well on visual reasoning, they remain fragile to viewpoint changes, as pixel-wise warping is highly sensitive to small depth errors and often introduces geometric distortions. Drawing on theories of mental imagery that posit part-level structural representations as the basis for human perspective transformation, we examine whether image tokens in ViT-based MLLMs serve as an effective substrate for viewpoint changes. We compare forward and backward warping, finding that backward token warping, which defines a dense grid on the target view and retrieves a corresponding source-view token for each grid point, achieves greater stability and better preserves semantic coherence under viewpoint shifts. Experiments on our proposed ViewBench benchmark demonstrate that token-level warping enables MLLMs to reason reliably from nearby viewpoints, consistently outperforming all baselines including pixel-wise warping approaches, spatially fine-tuned MLLMs, and a generative warping method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑