视觉语言模型能否胜任？通过MeasureBench评估视觉测量阅读
Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench
AI总结：
本文提出MeasureBench基准测试，评估视觉语言模型在测量阅读任务中的表现，发现即使最强模型也难以处理，通过强化微调提升性能，揭示了当前模型在细粒度空间定位上的局限。
摘要：
阅读测量仪器对人类来说轻而易举，但目前的视觉语言模型（VLMs）在这一任务上仍面临挑战。本文引入MeasureBench，涵盖真实和合成图像的多种测量类型，提供可扩展的数据合成流程。评估显示，即使最前沿的VLMs在测量阅读上也表现不佳。通过强化微调合成数据，在领域内合成子集和真实图像上均取得显著提升。分析指出当前VLMs在细粒度空间定位上的根本限制，希望此资源和代码能促进视觉基础数理和精确空间感知的未来进展，弥合识别数字与测量世界之间的差距。
英文摘要：
Reading measurement instruments is effortless for humans and requires relatively little domain expertise, yet it remains surprisingly challenging for current vision-language models (VLMs) as we find in preliminary evaluation. In this work, we introduce MeasureBench, a benchmark on visual measurement reading covering both real-world and synthesized images of various types of measurements, along with an extensible pipeline for data synthesis. Our pipeline procedurally generates a specified type of gauge with controllable visual appearance, enabling scalable variation in key details such as pointers, scales, fonts, lighting, and clutter. Evaluation on popular proprietary and open-weight VLMs shows that even the strongest frontier VLMs struggle with measurement reading in general. We have also conducted preliminary experiments with reinforcement finetuning (RFT) over synthetic data, and find a significant improvement on both in-domain synthetic subset and real-world images. Our analysis highlights a fundamental limitation of current VLMs in fine-grained spatial grounding. We hope this resource and our code releases can help future advances on visually grounded numeracy and precise spatial perception of VLMs, bridging the gap between recognizing numbers and measuring the world.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
