HANIA：规划器引导的多模态图证据选择用于基于证据的问答
HANIA: Planner-Guided Multimodal Graph Evidence Selection for Grounded Question Answering
浏览论文内容
中文总结 AI 辅助
本文提出HANIA框架，通过规划器引导的多模态图证据选择处理ScienceQA问答任务，无需目标数据集微调或迭代检索即可实现具竞争力的多模态问答。
中文摘要 AI 辅助
多模态问答仍易受噪声、不完整及弱对齐证据的影响，长且非结构化的上下文会引入冗余并催生无依据的生成，而扁平检索可能忽略多步推理所需的关系。本文提出HANIA，一种规划器引导的多模态图框架，用于基于证据的问答。HANIA使用冻结的视觉语言模型处理输入的图像和文本，提取与问题相关的简洁视觉证据，必要时执行弃权（不执行）操作；随后构建基于输入的多模态图，并应用两组有限状态规划器协调描述性与关系性证据。覆盖感知剪枝根据相关性、图置信度、概念覆盖度及模态多样性保留紧凑的证据集。最终将选定的段落、视觉陈述及图三元组输入冻结的指令调优解码器。我们在ScienceQA数据集上，以答案准确率、证据过滤质量、证据预算敏感性及效率为指标评估HANIA，结果表明，结构化证据规划与紧凑图引导检索可在无需目标数据集微调或迭代检索的情况下，实现具备竞争力的多模态问答。代码可在指定URL获取。
英文摘要
Multimodal question answering remains sensitive to noisy, incomplete, and weakly grounded evidence. Long unstructured contexts can introduce redundancy and encourage unsupported generation, while flat retrieval may overlook relations needed for multi-step reasoning. We present HANIA, a planner-guided multimodal graph framework for evidence-grounded question answering. HANIA processes the supplied image and text using a frozen vision-language model to extract concise question-relevant visual evidence with explicit abstention. It then constructs an input-grounded multimodal graph and applies a two-group finite-state planner to coordinate descriptive and relational evidence. Coverage-aware pruning retains a compact evidence set based on relevance, graph confidence, concept coverage, and modality diversity. The selected passages, visual statements, and graph triples are provided to a frozen instruction-tuned decoder. We evaluate HANIA on ScienceQA using answer accuracy, evidence-filtering quality, evidence-budget sensitivity, and efficiency. The results show that structured evidence planning and compact graph-guided retrieval can support competitive multimodal question answering without target-dataset fine-tuning or iterative retrieval. The code is available at https://github.com/Zafar-southeast/HANIA.
发表机构
- Southeast University（东南大学）
- Portland Institute（波特兰研究所）
- Aristotle University of Thessaloniki（亚里士多德大学）
- Shandong University（山东大学）
机构由 AI 辅助整理，请以论文原文为准。
