A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations
York University(约克大学);Dialpad Canada Inc.(Dialpad加拿大公司);Princess Nourah Bint Abdulrahman University(诺拉·宾特·阿卜杜勒拉赫曼公主大学);National Center for AI, Saudi Arabia(沙特阿拉伯国家人工智能中心);Nanyang Technological University(南洋理工大学);Royal Bank of Canada(加拿大皇家银行);Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所);Salesforce Research(Salesforce研究院)
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, M… 展开作者
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
Large Language Models (LLMs) have recently gained significant attention due to their remarkable capabilities in performing diverse tasks across various domains. However, a thorough evaluation of these models is crucial before deploying them in real-world applications to ensure they produce reliable performance. Despite the well-established importance of evaluating LLMs in the community, the complexity of the evaluation process has led to varied evaluation setups, causing inconsistencies in findings and interpretations. To address this, we systematically review the primary challenges and limitations causing these inconsistencies and unreliable evaluations in various steps of LLM evaluation. Based on our critical review, we present our perspectives and recommendations to ensure LLM evaluations are reproducible, reliable, and robust.
Where Should I Study? Biased Language Models Decide! Evaluating Fairness in LMs for Academic Recommendations
Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI)(负责任人工智能中心(CeRAI)、Wadhwani数据科学与人工智能学校(WSAI));Indian Institute of Technology Madras(印度理工学院马德拉斯学院)
RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation
The Hong Kong University of Science and Technology(香港科技大学);Southern University of Science and Technology(南方科技大学);The Chinese University of Hong Kong(香港中文大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学深圳校区);Shenzhen University General Hospital(深圳大学人民医院)
A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations
Microsoft Research Asia(微软亚洲研究院);Hong Kong University of Science and Technology(香港科技大学);Peking University(北京大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews
Keele University(基尔大学);Universidad de la República(共和国大学);Wrocław University of Science and Technology(弗罗茨瓦夫理工大学);University of Calgary(卡尔加里大学);Brunel University of London(伦敦布鲁内尔大学);Durham University(杜伦大学)
Evaluating TCFD Reporting: A New Application of Zero-Shot Analysis to Climate-Related Financial Disclosures
Justus Liebig University Giessen(吉森尤斯图斯·利比希大学);Justus Liebig University Giessen Centre for International Development and Environmental Research (ZEU)(吉森尤斯图斯·利比希大学国际发展与环境研究中心)
Evaluating Document Representations for Content-based Legal Literature Recommendations
Open Legal Data(开放法律数据);German Research Center for Artificial Intelligence(德国人工智能研究中心);ETH Zurich(苏黎世联邦理工学院);University of Wuppertal(伍珀塔尔大学)
推荐系统帮助法律专业人士找到相关文献来支持他们的案件。尽管它的重要性的专业,法律应用并没有反映在推荐系统和表示学习研究的最新进展。同时,法律推荐系统通常在没有任何公共基准数据集的情况下在小规模用户研究中进行评估。因此,这些研究的重现性有限。为了解决研究与实践之间的差距,我们探索了一套国家的最先进的文件表示方法检索语义相关的美国判例法的任务。我们评估基于文本的(例如,fastText、Transformers)、基于引用的(例如,DeepWalk,Poincaré)和混合方法。我们使用两种银标准对2,964份文件进行了注释,共比较了27种方法。银标准是从Open Case Book和Wikispecies新创建的,可以在开放许可下重复使用,以促进可重复性。我们的实验表明,来自平均fastText单词向量(在法律语料库上训练)的文档表示产生了最好的结果,紧随其后的是庞加莱引用嵌入。以混合方式组合fastText和Poincaré进一步改善了整体结果。除了整体性能外,我们还根据文档长度,引用计数和建议的覆盖范围分析了这些方法。我们在https://github.com/malteos/legal-document-similarity/上公开我们的源代码、模型和数据集。
Do recommender systems function in the health domain: a system review
School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);The 1st Affiliated Hospital of Harbin Medical University(哈尔滨医科大学附属第一医院)