近年来,大语言模型(Large Language Models, LLM)评估已成为热门研究领域。LLM评估的三个核心问题是“评估什么、在哪里评估、如何评估”。但现有研究主要聚焦前两个问题,基本围绕测试时给LLM分配什么任务、它需要处理哪类知识展开。而第三个问题涉及采用何种标准、评估者类型、评分方式及排名方法,目前相关讨论还较少。\n本文通过对比多种人工与自动评估标准开展评估方法分析,用到了现场标注、众包、公共标注者及GPT-4,结合了不同的评分方法与排名体系。我们提出了一个新数据集LLMEval,并对20个LLM开展了评估。共有2186名人员参与,最终生成243337条人工标注结果与57511条自动评估结果。我们对不同设置进行了对比分析,得出10条结论,可为未来的LLM评估工作提供一定参考。该数据集与评估结果已在https://github.com/llmeval 公开发布。
英文摘要:
Recently, the evaluation of Large Language Models has emerged as a popular area of research. The three crucial questions for LLM evaluation are ``what, where, and how to evaluate''. However, the existing research mainly focuses on the first two questions, which are basically what tasks to give the LLM during testing and what kind of knowledge it should deal with. As for the third question, which is about what standards to use, the types of evaluators, how to score, and how to rank, there hasn't been much discussion. In this paper, we analyze evaluation methods by comparing various criteria with both manual and automatic evaluation, utilizing onsite, crowd-sourcing, public annotators and GPT-4, with different scoring methods and ranking systems. We propose a new dataset, LLMEval and conduct evaluations on 20 LLMs. A total of 2,186 individuals participated, leading to the generation of 243,337 manual annotations and 57,511 automatic evaluation results. We perform comparisons and analyses of different settings and conduct 10 conclusions that can provide some insights for evaluating LLM in the future. The dataset and the results are publicly available at https://github.com/llmeval .
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
College of Computer Science and Artificial Intelligence, Fudan University(计算机科学与人工智能学院,复旦大学);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);Shanghai AI Laboratory(上海人工智能实验室);Northwestern University(西北大学)
Are Large Language Models Good Fact Checkers: A Preliminary Study
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
From Prompt Optimization to Multi-Dimensional Credibility Evaluation: Enhancing Trustworthiness of Chinese LLM-Generated Liver MRI Reports -- with Preliminary Extension to Lung Cancer
Yu-Yue Pathology Research Center, Jinfeng Laboratory, Chongqing, China(渝粤病理研究所,金风实验室,重庆,中国);T Magnetic Resonance Imaging Translational Medical Center, Department of Radiology, Southwest Hospital, Army Medical University, Chongqing, China(7T磁共振成像转化医学中心,放射科,西南医院,中国人民解放军军医大学,重庆,中国)
ADRD-Bench: A Preliminary LLM Benchmark for Alzheimer's Disease and Related Dementias
Electrical Engineering, University of Notre Dame(诺丁汉大学电气工程系);Computer Science and Engineering, University of Notre Dame(诺丁汉大学计算机科学与工程系);School of Medicine, Indiana University(印第安纳大学医学院)
Scaling Up Formal Representation of Clinical Trial Protocols in Ensemble Logic Using LLMs: A Preliminary Study
McGovern Medical School(麦戈文医学学院);McWilliams School of Biomedical Informatics(麦克威廉斯生物医学信息学学院);The University of Texas Health Science Center at Houston(德克萨斯大学健康科学中心休斯顿分校)
GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing
The Hong Kong University of Science & Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science & Technology(香港科技大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Zhejiang University(浙江大学);Great Bay University(大湾区大学);Hangzhou Research Institute, Beihang University(北京航空航天大学杭州研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)