RaTA-Tool: 基于检索的多模态大语言模型工具选择
RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models
AI总结：
本文提出RaTA-Tool框架，利用多模态大语言模型将多模态查询转化为结构化任务描述，并通过语义丰富的工具描述检索合适工具，支持开放世界场景下的工具扩展。
摘要：
工具学习旨在赋予AI系统调用外部资源的能力，如API、计算工具和专用模型，以解决超出单一语言生成能力的复杂任务。尽管近年来大语言模型和多模态大语言模型在推理和感知能力上有所提升，但现有工具使用方法大多局限于文本输入和封闭世界设置。因此，它们难以解释多模态用户指令，并且无法泛化到训练时未见过的工具。本文引入RaTA-Tool，一种新的开放世界多模态工具选择框架。与学习用户查询到固定工具标识符的直接映射不同，我们的方法使MLLM将多模态查询转换为结构化任务描述，并通过匹配此表示与语义丰富的、机器可读的工具描述来检索最合适的工具。这种基于检索的公式自然支持扩展到新工具而无需重新训练。为进一步提高任务描述与工具选择之间的对齐，我们结合基于偏好的优化阶段，使用直接偏好优化（DPO）。为了支持该领域的研究，我们还引入了第一个开放世界多模态工具使用的数据集，其工具描述源自Hugging Face模型卡片。广泛的实验表明，我们的方法显著提高了工具选择性能，特别是在开放世界、多模态场景中。
英文摘要：
Tool learning with foundation models aims to endow AI systems with the ability to invoke external resources -- such as APIs, computational utilities, and specialized models -- to solve complex tasks beyond the reach of standalone language generation. While recent advances in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have expanded their reasoning and perception capabilities, existing tool-use methods are predominantly limited to text-only inputs and closed-world settings. Consequently, they struggle to interpret multimodal user instructions and cannot generalize to tools unseen during training. In this work, we introduce RaTA-Tool, a novel framework for open-world multimodal tool selection. Rather than learning direct mappings from user queries to fixed tool identifiers, our approach enables an MLLM to convert a multimodal query into a structured task description and subsequently retrieve the most appropriate tool by matching this representation against semantically rich, machine-readable tool descriptions. This retrieval-based formulation naturally supports extensibility to new tools without retraining. To further improve alignment between task descriptions and tool selection, we incorporate a preference-based optimization stage using Direct Preference Optimization (DPO). To support research in this setting, we also introduce the first dataset for open-world multimodal tool use, featuring standardized tool descriptions derived from Hugging Face model cards. Extensive experiments demonstrate that our approach significantly improves tool-selection performance, particularly in open-world, multimodal scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
