TRACER:多模态工具使用代理的可验证生成溯源
TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents
AI总结:
TRACER通过生成带结构溯源记录的答案句子,解决多模态工具使用代理中缺乏的 claim-level 依赖结构问题,提升工具使用可验证性和优化性。
摘要:
多模态大语言模型通过调用外部工具解决以视觉为中心的任务,但当前工具使用代理通常只暴露执行的工具轨迹和最终答案,而很少明确哪些工具观察支持每个生成的声明。我们称这种缺失的声明级依赖结构为溯源缺口。该缺口使工具使用难以验证和优化,因为有用的证据、冗余探索和不支持的推理混合在同一轨迹中。我们介绍了TRACER,一个用于多模态工具使用代理的可验证生成溯源框架。TRACER不是在生成后添加引用,而是生成每个答案句子的同时生成结构化的溯源记录,该记录标识支持的工具步骤、证据单元和语义支持关系。其关系空间包含引用、压缩和推理,涵盖直接重用、忠实压缩和基于事实的推导。TRACER通过模式检查、工具步骤对齐、来源真实性以及关系合理性验证每个记录,然后将验证的溯源转换为可追溯约束和基于强化学习的溯源衍生局部信用。我们进一步构建了TRACE-Bench,一个从粗粒度多模态工具轨迹中重建句子级溯源的基准。在TRACE-Bench上,简单添加工具往往会引入噪声。使用Qwen3-VL-8B,TRACER达到78.23%的答案准确率和95.72%的摘要准确率,比最强的封闭源工具增强基线高出23.80个百分点。与仅工具监督微调相比,它还减少了总测试集工具调用次数从4949次到3486次。这些结果表明,可靠的多模态工具推理依赖于对观察的溯源感知使用,而非单纯增加工具调用次数。
英文摘要:
Multimodal large language models increasingly solve vision-centric tasks by calling external tools for visual inspection, OCR, retrieval, calculation, and multi-step reasoning. Current tool-using agents usually expose the executed tool trajectory and the final answer, but they rarely specify which tool observation supports each generated claim. We call this missing claim-level dependency structure the provenance gap. The gap makes tool use hard to verify and hard to optimize, because useful evidence, redundant exploration, and unsupported reasoning are mixed in the same trajectory. We introduce TRACER, a framework for verifiable generative provenance in multimodal tool-using agents. Instead of adding citations after generation, TRACER generates each answer sentence together with a structured provenance record that identifies the supporting tool turn, evidence unit, and semantic support relation. Its relation space contains Quotation, Compression, and Inference, covering direct reuse, faithful condensation, and grounded derivation. TRACER verifies each record through schema checking, tool-turn alignment, source authenticity, and relation rationality, and then converts verified provenance into traceability constraints and provenance-derived local credit for reinforcement learning. We further construct TRACE-Bench, a benchmark for sentence-level provenance reconstruction from coarse multimodal tool trajectories. On TRACE-Bench, simply adding tools often introduces noise. With Qwen3-VL-8B, TRACER reaches 78.23% answer accuracy and 95.72% summary accuracy, outperforming the strongest closed-source tool-augmented baseline by 23.80 percentage points. Compared with tool-only supervised fine-tuning, it also reduces total test-set tool calls from 4949 to 3486. These results show that reliable multimodal tool reasoning depends on provenance-aware use of observations, not on more tool calls alone.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。