Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
KAIST(韩国科学技术院);LG AI Research(LG人工智能研究院);Carnegie Mellon University(卡内基梅隆大学);MIT(麻省理工学院);Allen Institute for AI(艾伦人工智能研究所);University of Illinois Chicago(芝加哥伊利诺伊大学)
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs. However, concerns including transparency, controllability, and affordability strongly motivate the development of open-source LMs specialized in evaluations. On the other hand, existing open evaluator LMs exhibit critical shortcomings: 1) they issue scores that significantly diverge from those assigned by humans, and 2) they lack the flexibility to perform both direct assessment and pairwise ranking, the two most prevalent forms of assessment. Additionally, they do not possess the ability to evaluate based on custom evaluation criteria, focusing instead on general attributes like helpfulness and harmlessness. To address these issues, we introduce Prometheus 2, a more powerful evaluator LM than its predecessor that closely mirrors human and GPT-4 judgements. Moreover, it is capable of processing both direct assessment and pair-wise ranking formats grouped with a user-defined evaluation criteria. On four direct assessment benchmarks and four pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and agreement with humans and proprietary LM judges among all tested open evaluator LMs. Our models, code, and data are all publicly available at https://github.com/prometheus-eval/prometheus-eval.
Evaluating LLMs on Generating Age-Appropriate Child-Like Conversations
Department of Holistic Systems, SimulaMet(整体系统系,SimulaMet);Department of Computer Science, Oslo Metropolitan University(计算机科学系,奥斯陆 Metropolitan 大学);Department of Behavioural Sciences, Oslo Metropolitan University(行为科学系,奥斯陆 Metropolitan 大学);Department of Psychology, Harvard University(心理学系,哈佛大学);Department of Informatics, University of Oslo(信息学系,奥斯陆大学);Norwegian Computing Center(挪威计算中心)
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
NLP;CT Lab, Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门大学CT实验室);Department of Portuguese, Faculty of Arts and Humanities, University of Macau(人文学院葡萄牙语系,澳门大学)
JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
Kyoto University(京都大学);Hohai University(河海大学);The University of Tokyo(东京大学);University of Science and Technology of China(中国科学技术大学);Hong Kong Polytechnic University(香港理工大学)
TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health
Renmin University of China, China(中国人民大学);Beijing University of Posts and Telecommunications, China(北京邮电大学);Hefei University of Technology, China(合肥工业大学)
ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Centre for Medical Informatics, The University of Edinburgh(爱丁堡大学医学信息学中心);Health Data Research UK(英国健康数据研究所);School of Computer Science, Peking University(北京大学计算机学院);National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);ETH Zurich(苏黎世联邦理工学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院)
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能实验室);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海教育人工智能研究院)
Can Agents Judge Systematic Reviews Like Humans? Evaluating SLRs with LLM-based Multi-Agent System
Department of Computer Science(计算机科学系);Information Technology University(信息技术大学);AI Center for Precision Health(精准健康人工智能中心);Weill Cornell Medicine–Qatar(韦尔·科恩医学中心–卡塔尔);Computer Science and Engineering Department(计算机科学与工程系)