MLLM-Tool：一种用于工具智能体学习的多模态大语言模型
MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning
AI总结：
本文提出MLLM-Tool系统，融合开源大语言模型与多模态编码器，使模型能感知多模态指令并正确选择匹配工具，同时构建含多候选选择的数据集以评估工具推荐能力。
摘要：
近年来，大语言模型（LLMs）在自然语言理解和生成任务中展现出的惊人性能，引发了大量将其作为中央控制器来构建智能体系统的探索。多项研究致力于将LLMs与外部工具连接，以扩展应用场景。然而，当前LLMs感知工具使用的能力仅限于单一文本查询，这可能导致在理解用户真实意图时产生歧义。LLMs被期望通过感知视觉或听觉基础指令中的信息来消除这种歧义。因此，在本文中，我们提出了MLLM-Tool，一个融合开源LLMs和多模态编码器的系统，使学习后的LLMs能够感知多模态输入指令，并正确选择功能匹配的工具。为便于评估模型能力，我们从HuggingFace收集了一个以多模态输入工具为特征的数据集。我们数据集的另一个重要特点是，由于相同功能和同义功能的存在，同一条指令也包含多个潜在选择，这为同一查询提供了更多潜在解决方案。实验表明，我们的MLLM-Tool能够为多模态指令推荐合适的工具。代码和数据可在 https://github.com/MLLM-Tool/MLLM-Tool 获取。
英文摘要：
Recently, the astonishing performance of large language models (LLMs) in natural language comprehension and generation tasks triggered lots of exploration of using them as central controllers to build agent systems. Multiple studies focus on bridging the LLMs to external tools to extend the application scenarios. However, the current LLMs' ability to perceive tool use is limited to a single text query, which may result in ambiguity in understanding the users' real intentions. LLMs are expected to eliminate that by perceiving the information in the visual- or auditory-grounded instructions. Therefore, in this paper, we propose MLLM-Tool, a system incorporating open-source LLMs and multi-modal encoders so that the learned LLMs can be conscious of multi-modal input instruction and then select the function-matched tool correctly. To facilitate the evaluation of the model's capability, we collect a dataset featuring multi-modal input tools from HuggingFace. Another essential feature of our dataset is that it also contains multiple potential choices for the same instruction due to the existence of identical functions and synonymous functions, which provides more potential solutions for the same query. The experiments reveal that our MLLM-Tool is capable of recommending appropriate tools for multi-modal instructions. Codes and data are available at https://github.com/MLLM-Tool/MLLM-Tool.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
