arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.07398cs.AIcs.CL
收藏

LLMEval:关于如何评估大语言模型的初步研究

LLMEval: A Preliminary Study on How to Evaluate Large Language Models

Fudan University(复旦大学);Shanghai Jiaotong University(上海交通大学)

Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, Xuanjing Huang

AI总结:

该研究针对大语言模型评估中“如何评估”这一少被探讨的问题,对比多类人工与自动评估标准,构建LLMEval数据集并对20个LLM开展评估,得出10条可为后续评估提供参考的结论,相关资源已公开。

摘要:

近年来,大语言模型(Large Language Models, LLM)评估已成为热门研究领域。LLM评估的三个核心问题是“评估什么、在哪里评估、如何评估”。但现有研究主要聚焦前两个问题,基本围绕测试时给LLM分配什么任务、它需要处理哪类知识展开。而第三个问题涉及采用何种标准、评估者类型、评分方式及排名方法,目前相关讨论还较少。\n本文通过对比多种人工与自动评估标准开展评估方法分析,用到了现场标注、众包、公共标注者及GPT-4,结合了不同的评分方法与排名体系。我们提出了一个新数据集LLMEval,并对20个LLM开展了评估。共有2186名人员参与,最终生成243337条人工标注结果与57511条自动评估结果。我们对不同设置进行了对比分析,得出10条结论,可为未来的LLM评估工作提供一定参考。该数据集与评估结果已在https://github.com/llmeval 公开发布。

英文摘要:

Recently, the evaluation of Large Language Models has emerged as a popular area of research. The three crucial questions for LLM evaluation are ``what, where, and how to evaluate''. However, the existing research mainly focuses on the first two questions, which are basically what tasks to give the LLM during testing and what kind of knowledge it should deal with. As for the third question, which is about what standards to use, the types of evaluators, how to score, and how to rank, there hasn't been much discussion. In this paper, we analyze evaluation methods by comparing various criteria with both manual and automatic evaluation, utilizing onsite, crowd-sourcing, public annotators and GPT-4, with different scoring methods and ranking systems. We propose a new dataset, LLMEval and conduct evaluations on 20 LLMs. A total of 2,186 individuals participated, leading to the generation of 243,337 manual annotations and 57,511 automatic evaluation results. We perform comparisons and analyses of different settings and conduct 10 conclusions that can provide some insights for evaluating LLM in the future. The dataset and the results are publicly available at https://github.com/llmeval .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑