CIBench:使用代码解释器插件评估你的大语言模型
CIBench: Evaluating Your LLMs with a Code Interpreter Plugin
AI总结:
本文提出CIBench交互式评估框架,含数据集和两种模式,全面评估24个大语言模型在数据科学任务中利用代码解释器的能力,并提供未来改进见解。
摘要:
虽然基于大语言模型的智能体(它们使用外部工具来解决复杂问题)已经取得了显著进展,但对它们的能力进行基准测试具有挑战性,从而阻碍了对其局限性的清晰理解。在本文中,我们提出了一个名为CIBench的交互式评估框架,以全面评估大语言模型在数据科学任务中利用代码解释器的能力。我们的评估框架包括一个评估数据集和两种评估模式。该评估数据集采用大语言模型与人类合作的方式构建,并通过利用连续且交互式的IPython会话模拟真实工作流程。两种评估模式分别评估大语言模型在有和没有人类辅助下的能力。我们进行了大量实验,分析了24个大语言模型在CIBench上的表现,并为未来大语言模型在代码解释器利用方面提供了有价值的见解。
英文摘要:
While LLM-Based agents, which use external tools to solve complex problems, have made significant progress, benchmarking their ability is challenging, thereby hindering a clear understanding of their limitations. In this paper, we propose an interactive evaluation framework, named CIBench, to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks. Our evaluation framework includes an evaluation dataset and two evaluation modes. The evaluation dataset is constructed using an LLM-human cooperative approach and simulates an authentic workflow by leveraging consecutive and interactive IPython sessions. The two evaluation modes assess LLMs' ability with and without human assistance. We conduct extensive experiments to analyze the ability of 24 LLMs on CIBench and provide valuable insights for future LLMs in code interpreter utilization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。