arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00104cs.CVcs.AI
收藏

R3G: 一种面向以视觉为中心的答案生成的推理-检索-重排序框架

R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation

The Shenzhen International Graduate School, Tsinghua University, China(清华大学深圳国际研究生院);State Key Laboratory of Nuclear Power Safety Technology and Equipment, China(核能安全技术与装备国家重点实验室);School of Computer Science and Information Engineering, Hefei University of Technology, China(合肥工业大学计算机科学与信息工程学院)

Zhuohong Chen, Zhengxian Wu, Zirui Liao, Shenao Jiang, Hangrui Xu, Yang Chen, Chaokui Su, Xiaoyu Liu, Haoqian Wang

AI总结:

提出R3G框架,通过先制定推理计划指定所需视觉线索,再采用粗检索加细粒度重排序的两阶段策略选择证据图像,在MRAG-Bench上提升六种多模态大语言模型在九个子场景中的准确率,实现整体最优性能。

摘要:

以视觉为中心的VQA检索需要检索图像以补充缺失的视觉线索,并将其整合到推理过程中。然而,选择正确的图像并将其有效整合到模型的推理中仍然具有挑战性。为了解决这一挑战,我们提出了R3G,一个模块化的推理-检索-重排序框架。它首先生成一个简要的推理计划,指定所需的视觉线索,然后采用两阶段策略,先进行粗检索,再进行细粒度重排序,以选择证据图像。在MRAG-Bench上,R3G在六个多模态大语言模型骨干和九个子场景中提高了准确率,实现了整体最优性能。消融实验表明,充分性感知的重排序和推理步骤是互补的,有助于模型既选择正确的图像又充分利用它们。我们在https://github.com/czh24/R3G发布代码和数据。

英文摘要:

Vision-centric retrieval for VQA requires retrieving images to supply missing visual cues and integrating them into the reasoning process. However, selecting the right images and integrating them effectively into the model's reasoning remains challenging. To address this challenge, we propose R3G, a modular Reasoning-Retrieval-Reranking framework. It first produces a brief reasoning plan that specifies the required visual cues, then adopts a two-stage strategy, with coarse retrieval followed by fine-grained reranking, to select evidence images. On MRAG-Bench, R3G improves accuracy across six MLLM backbones and nine sub-scenarios, achieving state-of-the-art overall performance. Ablations show that sufficiency-aware reranking and reasoning steps are complementary, helping the model both choose the right images and use them well. We release code and data at https://github.com/czh24/R3G.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑