arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.16932cs.CVcs.AI
收藏

关注关键区域:为高效视觉语言模型的高分辨率作物检索

Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs

IBM Research(IBM研究院);Tel-Aviv University(特拉维夫大学);Technion(技术学院);Ben-Gurion University(本· Gurion大学)

Nimrod Shabtay, Moshe Kimhi, Artem Spector, Sivan Haray, Ehud Rivlin, Chaim Baskin, Raja Giryes, Eli Schwartz

AI总结:

AwaRes通过低分辨率全局视图和工具调用实现准确与效率的平衡,自动构建监督数据并结合复合奖励训练框架,提升视觉语言模型的检索效率。

摘要:

视觉语言模型(VLMs)通常以原生高分辨率处理图像,导致准确性和计算效率之间的权衡:高分辨率输入能捕捉细节但计算成本高,而低分辨率输入虽高效但可能遗漏关键信息,如小文本。我们提出AwaRes,一种按需空间框架,通过低分辨率全局视图和工具调用检索所需高分辨率区域,解决此权衡。我们自动构建监督数据:裁判比较低/高分辨率答案以标记是否需要裁剪, oracle接地模型定位正确答案的证据,映射为离散裁剪集形成多轮工具使用轨迹。我们通过冷启动SFT和多轮GRPO训练框架,结合语义答案正确性和显式裁剪成本惩罚的复合奖励。

英文摘要:

Vision-language models (VLMs) typically process images at a native high-resolution, forcing a trade-off between accuracy and computational efficiency: high-resolution inputs capture fine details but incur significant computational costs, while low-resolution inputs advocate for efficiency, they potentially miss critical visual information, like small text. We present AwaRes, a spatial-on-demand framework that resolves this accuracy-efficiency trade-off by operating on a low-resolution global view and using tool-calling to retrieve only high-resolution segments needed for a given query. We construct supervised data automatically: a judge compares low- vs.\ high-resolution answers to label whether cropping is needed, and an oracle grounding model localizes the evidence for the correct answer, which we map to a discrete crop set to form multi-turn tool-use trajectories. We train our framework with cold-start SFT followed by multi-turn GRPO with a composite reward that combines semantic answer correctness with explicit crop-cost penalties. Project page: https://nimrodshabtay.github.io/AwaRes

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑