PGT: 用于提升多模态大语言模型视觉定位的程序化生成任务
PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs
AI总结：
提出程序化生成任务(PGT)框架，通过叠加几何基元生成密集监督信号，解耦视觉定位能力与语义先验，显著提升多模态大语言模型在细粒度理解任务上的表现。
摘要：
尽管多模态大语言模型(MLLMs)取得了显著进展，但这些模型在细粒度理解任务上仍然存在困难。在这项工作中，我们提出了程序化生成任务(PGT)，一个简单的数据驱动框架，具有双重目的：诱导细粒度视觉理解，并作为低成本的诊断工具来识别感知失败的来源。通过在图像上叠加明确的几何基元，PGT生成额外的密集监督，将视觉定位能力与语义先验解耦。在关系、定量和3D/深度理解基准上的大量实验表明，PGT在各种架构上均取得了显著提升。在使用PGT数据增强的LLaVA-v1.5-Instruct上进行指令微调，在What'sUp基准上提升高达+20%，在CV-Bench-2D上提升+13.3%，同时保持通用感知能力。此外，在PGT数据上微调最先进的MLLMs，在What'sUp上提升高达+5.5%，在CV-Bench-2D上提升+8.3%。这些发现表明，PGT有效解决了细粒度感知的瓶颈，揭示了许多空间推理缺陷源于监督信号不足，而非固有的架构或分辨率限制。
英文摘要：
Despite remarkable progress in Multimodal Large Language Models (MLLMs), these models still struggle with fine-grained understanding tasks. In this work, we propose Procedurally Generated Tasks (PGT), a simple data-driven framework that serves a dual purpose: inducing fine-grained visual understanding and acting as a low-cost diagnostic tool to identify the source of perception failures. By overlaying unambiguous geometric primitives on images, PGT generate additional dense supervision that disentangles visual grounding capability from semantic priors. Extensive experiments on relational, quantitative, and 3D/depth understanding benchmarks show that PGT yields remarkable gains across diverse architectures. Instruction tuning MLLMs on LLaVA-v1.5-Instruct augmented with PGT data results in improvements of up to +20% on the What'sUp benchmark and +13.3% on CV-Bench-2D, while maintaining general perception capabilities. Moreover, finetuning state-of-the-art MLLMs on PGT data leads to boosts of up to +5.5% on What'sUp and +8.3% on CV-Bench-2D. These findings demonstrate that PGT effectively address the bottleneck of fine-grained perception, revealing that many spatial reasoning deficits stem from inadequate supervision signals rather than inherent architectural or resolution limitations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
