arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11044cs.CLcs.AIcs.LG
收藏

通过辩论评估大型语言模型的性能

Evaluating the Performance of Large Language Models via Debates

University of Pennsylvania(宾夕法尼亚大学)

Behrad Moniri, Hamed Hassani, Edgar Dobriban

AI总结:

本文提出以LLM辩论并由LLM裁判的自动化评估框架,在无需人工众包的情况下评估模型的知识与论证能力,所得排名与人工评测高度一致。

摘要:

大型语言模型(LLMs)正在快速发展并影响众多领域,因此亟需开发有效的方法来评估和比较它们的性能。当前大多数性能评估方法要么基于固定的、特定领域的问题,缺乏许多实际应用所需的灵活性;要么依赖人工输入,导致难以扩展。为解决这些问题,我们提出了一种基于LLMs之间辩论并由另一个LLM担任裁判的自动化基准测试框架。该方法不仅评估领域知识,还评估论证推理和识别不一致性等技能。我们使用该辩论框架评估了多种最先进的LLMs的性能,并获得了与基于人工输入的流行排名高度一致的排名,从而消除了对昂贵的人工众包的需求。

英文摘要:

Large Language Models (LLMs) are rapidly evolving and impacting various fields, necessitating the development of effective methods to evaluate and compare their performance. Most current approaches for performance evaluation are either based on fixed, domain-specific questions that lack the flexibility required in many real-world applications, or rely on human input, making them unscalable. To address these issues, we propose an automated benchmarking framework based on debates between LLMs, judged by another LLM. This method assesses not only domain knowledge, but also skills such as argumentative reasoning and inconsistency recognition. We evaluate the performance of various state-of-the-art LLMs using the debate framework and achieve rankings that align closely with popular rankings based on human input, eliminating the need for costly human crowdsourcing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑