arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21517cs.CV
收藏

我应该信任哪个工具响应?具有工具-专业知识意识的胸片代理与多模态代理学习

Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning

The Hong Kong University of Science(香港科学与技术大学)

Zheang Huai, Honglong Yang, Xiaomeng Li

AI总结:

本文提出TEA-CXA,一种具有工具-专业知识意识的胸片代理,通过多模态代理学习解决医疗工具冲突问题,提升多轮工具调用的可靠性。

摘要:

具有工具使用能力的AI代理在整合各种工具的领域专业知识方面显示出前景。然而在医疗领域,工具通常是固有易错的AI模型,可能产生矛盾的响应。现有医疗代理研究对工具的现实可靠性理解不足,因此无法有效解决工具冲突。为了解决这一差距,本文介绍了一个框架,使代理能够与工具交互,并通过代理学习实证学习不同类型的多模态查询的实用可信度。作为具体实现,我们专注于胸片分析,并提出一个具有工具-专业知识意识的胸片代理(TEA-CXA)。当工具输出矛盾时,代理实验性地接受或拒绝多模态工具结果,接收奖励,并学习每种查询类型应信任哪个工具。重要的是,TEA-CXA扩展了现有的强化学习代码库,以支持文本输入的多轮工具调用,以有效支持多模态上下文。此外,我们通过支持每轮中多个工具调用、并行工具推断和单个用户查询中的多图像适应来增强医疗使用场景的代码库。我们的代码框架适用于多轮工具调用强化学习在多模态设置中的通用医疗研究。实验表明,TEA-CXA优于最先进的方法和全面的基线集合。代码将被发布。

英文摘要:

AI agents with tool-use capabilities show promise for integrating the domain expertise of various tools. In the medical field, however, tools are usually AI models that are inherently error-prone and can produce contradictory responses. Existing research on medical agents lacks sufficient understanding of the tools' realistic reliability and thus cannot effectively resolve tool conflicts. To address this gap, this paper introduces a framework that enables an agent to interact with tools and empirically learn their practical trustworthiness across different types of multimodal queries via agentic learning. As a concrete instantiation, we focus on chest X-ray analysis and present a tool-expertise-aware chest X-ray agent (TEA-CXA). When tool outputs disagree, the agent experimentally accepts or rejects multimodal tool results, receives rewards, and learns which tool to trust for each query type. Importantly, TEA-CXA extends existing codebases for reinforcement learning with multi-turn tool-calling that focus on textual inputs, to support multimodal contexts effectively. In addition, we enhance the codebase for medical use scenarios by supporting multiple tool calls in one turn, parallel tool inference, and multi-image accommodation within a single user query. Our code framework is applicable to general medical research on multi-turn tool-calling reinforcement learning in multimodal settings. Experiments show that TEA-CXA outperforms the state-of-the-art methods and a comprehensive set of baselines. Code will be released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑