arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.15955cs.LGcs.AI
收藏

大型语言模型在处理工具输出方面有多好?

How Good Are LLMs at Processing Tool Outputs?

IBM Research(IBM研究院);Persistent Systems;Grab

Kiran Kate, Yara Rizk, Poulami Ghosh, Ashu Gulati, Tathagata Chakraborti, Zidane Wright, Mayank Agarwal

AI总结:

本文研究了LLMs处理工具输出的能力,发现即使在前沿模型上,JSON处理仍具挑战性,不同提示策略影响性能差异达3%-50%。

摘要:

大多数现实任务自动化问题需要大型语言模型(LLMs)调用工具,这些工具往往返回复杂的JSON响应。这些响应必须进一步处理,以推导出完成任务所需的必要信息。LLMs在这方面的能力研究不足。在本文中,我们研究了工具响应处理任务以及LLMs处理结构化(JSON)响应的能力。我们为此任务创建了一个数据集,并使用多种提示方法评估了15个开源和闭源模型。我们的结果表明,即使在前沿模型上,JSON处理仍然是一个困难的任务,跨多种提示策略。最优的响应处理策略取决于工具输出的性质和大小,以及所需推理的复杂性。处理方法的差异可能导致性能差异,范围从3%到50%。

英文摘要:

Most realistic task automation problems require large language models (LLMs) to call tools, which often return complex JSON responses. These responses must be further processed to derive the information necessary for task completion. The ability of LLMs to do so is under-studied. In this paper, we study the tool response processing task and LLMs' abilities to process structured (JSON) responses. We created a dataset for this task, and evaluated 15 open and closed weight models using multiple prompting approaches. Our results show that JSON processing remains a difficult task even for frontier models across multiple prompting strategies. The optimal response processing strategy depends on both the nature and size of the tool outputs, as well as the complexity of the required reasoning. Variations in processing approaches can lead to performance differences ranging from 3\% to 50\%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑