arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.04069cs.CLcs.AIcs.LG
收藏

关于评估大型语言模型的系统综述与批判性评论:挑战、局限与建议

A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations

York University(约克大学);Dialpad Canada Inc.(Dialpad加拿大公司);Princess Nourah Bint Abdulrahman University(诺拉·宾特·阿卜杜勒拉赫曼公主大学);National Center for AI, Saudi Arabia(沙特阿拉伯国家人工智能中心);Nanyang Technological University(南洋理工大学);Royal Bank of Canada(加拿大皇家银行);Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所);Salesforce Research(Salesforce研究院)
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, M… 展开作者

Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang

AI总结:

本文系统梳理了LLM评估各环节中导致结果不一致和不可靠的主要挑战与局限,并提出确保评估可复现、可靠且稳健的观点与建议。

摘要:

大型语言模型(LLMs)近来因其在多个领域执行多样化任务时的卓越能力而受到广泛关注。然而,在将其部署到实际应用之前,对这些模型进行全面评估至关重要,以确保其能够产生可靠的性能。尽管评估 LLMs 在社区中的重要性已得到充分认可,但评估过程的复杂性导致了多种不同的评估设置,从而造成研究结果和解释上的不一致。为解决这一问题,我们系统性地梳理了在 LLM 评估各环节中导致这些不一致和不可靠评估的主要挑战与局限。基于我们的批判性评论,我们提出了自己的观点和建议,以确保 LLM 评估具有可复现性、可靠性和稳健性。

英文摘要:

Large Language Models (LLMs) have recently gained significant attention due to their remarkable capabilities in performing diverse tasks across various domains. However, a thorough evaluation of these models is crucial before deploying them in real-world applications to ensure they produce reliable performance. Despite the well-established importance of evaluating LLMs in the community, the complexity of the evaluation process has led to varied evaluation setups, causing inconsistencies in findings and interpretations. To address this, we systematically review the primary challenges and limitations causing these inconsistencies and unreliable evaluations in various steps of LLM evaluation. Based on our critical review, we present our perspectives and recommendations to ensure LLM evaluations are reproducible, reliable, and robust.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑