arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.01023cs.SE
收藏

通过多AI代理评估大型语言模型:初步结果

Large Language Model Evaluation Via Multi AI Agents: Preliminary results

Tampere University(坦佩雷大学);Jyväskylä University(于韦斯屈莱大学)

Zeeshan Rasheed, Muhammad Waseem, Kari Systä, Pekka Abrahamsson

AI总结:

本文提出多AI代理模型评估不同LLM性能,通过代码检索与验证,初步结果显示GPT-3.5 Turbo表现更优,未来将引入MBPP基准提升评估精度。

摘要:

随着大型语言模型(LLMs)在研究和日常操作中变得不可或缺,严格的评估至关重要。本文引入了一种新型多代理AI模型,旨在评估和比较不同LLM的性能。该模型包含八个不同AI代理,每个代理根据共同描述从不同先进语言模型中检索代码,包括GPT-3.5、GPT-3.5 Turbo、GPT-4、GPT-4 Turbo、Google Bard、LLAMA和Hugging Face。我们的模型利用每个语言模型的API来检索给定的高层描述的代码。此外,我们开发了一个验证代理,负责评估其同僚生成的代码。我们整合了HumanEval基准到验证代理中,以评估生成代码的性能,提供关于各自能力和效率的见解。初步结果表明,GPT-3.5 Turbo模型的性能相对优于其他模型。此初步分析作为基准,比较其性能。我们的未来目标是通过引入大规模多任务Python基准(MBPP)进一步细化评估。此外,我们计划与来自不同背景的二十名从业者分享我们的模型,以测试模型并收集反馈以进一步改进。

英文摘要:

As Large Language Models (LLMs) have become integral to both research and daily operations, rigorous evaluation is crucial. This assessment is important not only for individual tasks but also for understanding their societal impact and potential risks. Despite extensive efforts to examine LLMs from various perspectives, there is a noticeable lack of multi-agent AI models specifically designed to evaluate the performance of different LLMs. To address this gap, we introduce a novel multi-agent AI model that aims to assess and compare the performance of various LLMs. Our model consists of eight distinct AI agents, each responsible for retrieving code based on a common description from different advanced language models, including GPT-3.5, GPT-3.5 Turbo, GPT-4, GPT-4 Turbo, Google Bard, LLAMA, and Hugging Face. Our developed model utilizes the API of each language model to retrieve code for a given high-level description. Additionally, we developed a verification agent, tasked with the critical role of evaluating the code generated by its counterparts. We integrate the HumanEval benchmark into our verification agent to assess the generated code's performance, providing insights into their respective capabilities and efficiencies. Our initial results indicate that the GPT-3.5 Turbo model's performance is comparatively better than the other models. This preliminary analysis serves as a benchmark, comparing their performances side by side. Our future goal is to enhance the evaluation process by incorporating the Massively Multitask Benchmark for Python (MBPP) benchmark, which is expected to further refine our assessment. Additionally, we plan to share our developed model with twenty practitioners from various backgrounds to test our model and collect their feedback for further improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑