模态相关性并非模态效用：用于成本感知多模态RAG的事后选择性模态升级
Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG
AI总结：
研究多模态检索增强生成中模态选择决策点问题，提出事后选择性模态升级方法，先低成本从文本和表格回答，再按需支付VLM证据费用，校准路由器决定是否升级，在MultiModalQA上减少视觉调用并缩小与神谕升级率差距，扩展了路由信号层次结构。
摘要：
多模态检索增强生成（RAG）基于从异构模态（文本、表格和图像）中提取的证据来生成内容。目前主要的部署选择是二元的且在模型尝试回答之前就做出：要么运行低成本的文本（+表格）管道，要么为每个图像支付昂贵的视觉语言模型（VLM）费用。近期的自适应系统通过从问题条件预测器中选择模态或保真度来改进这一点。但研究表明这是错误的决策点。通过对MultiModalQA的神谕余量分析发现，模态与问题的相关性并不能很好预测是否真的需要该模态来正确回答。为此提出了事后选择性模态升级，先从文本和表格中低成本回答，运行验证器定位缺失模态，仅在需要时支付VLM证据费用。校准后的升级价值路由器决定预期的准确性提升是否能证明视觉成本的合理性。在MultiModalQA上，该路由器在发出更少视觉调用的情况下恢复了始终开启VLM管道的准确性，并缩小了与神谕升级率的差距。结果将为检索深度和推理跳数建立的路由信号层次结构扩展到了第三个轴——模态，在单一成本感知选择性升级视图下。
英文摘要：
Multimodal retrieval-augmented generation (RAG) grounds a generator in evidence drawn from heterogeneous modalities -- text, tables, and images. The dominant deployment choice is binary and made before the model has tried to answer: either run a cheap text(+table) pipeline, or pay for an expensive vision-language model (VLM) over every image. Recent adaptive systems improve on this by selecting the modality or fidelity pre-retrieval, from a question-conditioned predictor of which modality will be needed. We show that this is the wrong decision point. Through an oracle headroom analysis on MultiModalQA, we find that the relevance of a modality to a question is a weak predictor of whether that modality is actually needed to answer correctly: a large fraction of questions whose gold support includes an image are nonetheless answerable from text and tables alone, and a pre-retrieval router that escalates on apparent visual relevance over-escalates substantially relative to an oracle. We propose \textbf{post-hoc selective modality escalation}: answer cheaply from text and tables, run a verifier on the (query, draft answer, evidence) tuple that localizes which modality is missing, and pay for VLM evidence only there. A calibrated value-of-escalation router then decides whether the expected accuracy gain justifies the visual cost. On MultiModalQA, our router recovers the accuracy of an always-on VLM pipeline while issuing far fewer visual calls, and closes most of the gap to the oracle escalation rate. The result extends a routing-signal hierarchy established for retrieval depth and reasoning hops to a third axis -- modality -- under a single cost-aware selective-escalation view.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
