Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality. Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics. To address this, we propose QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which evaluates both generated questions and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. We demonstrate the appropriateness of these dimensions by examining their correlations and distinctions. Through consistent evaluations of QG models and automatic metrics with QGEval, we find that 1) most QG models perform unsatisfactorily in terms of answerability and answer consistency, and 2) existing metrics fail to align well with human judgments when evaluating generated questions across the 7 dimensions. We expect this work to foster the development of both QG technologies and their evaluation.
University of Bologna, Bologna, Italy(博洛尼亚大学);CNR - Institute of Cognitive Sciences and Technologies(意大利认知科学与技术研究院);University of Pisa, Pisa, Italy(比萨大学)
LEXam: Benchmarking Legal Reasoning on 340 Law Exams
ETH Zurich(苏黎世联邦理工学院);University of Zurich(苏黎世大学);University of Lausanne(洛桑大学);Max Planck Institute for Research on Collective Goods(马克斯·普朗克集体物品研究所);Omnilex;University of St. Gallen(圣加仑大学);Swiss Federal Supreme Court(瑞士联邦最高法院)
Benchmarking Interaction, Beyond Policy: a Reproducible Benchmark for Collaborative Instance Object Navigation
Sapienza University of Rome(罗马大学);University of Verona(维罗纳大学);Polytechnic of Turin(都灵理工大学);Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会);Reykjavik University(雷克雅未克大学)
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Microsoft(微软);University College London(伦敦大学学院);University of Illinois, UC(伊利诺伊大学厄巴纳-香槟分校);University of Waterloo(滑铁卢大学)
TREC,CLEF,NTCIR和FIRE等评估工作,以及MS MARCO等公共排行榜,旨在鼓励研究和跟踪我们的进展,解决我们领域的重大问题。然而,目标不仅仅是确定哪个运行是“最好的”,获得最高分数。我们的目标是通过开发新的强大的技术来推动该领域的发展,这些技术在许多不同的环境中工作,并在研究和实践中采用。本文使用MS MARCO和TREC深度学习Track作为我们的案例研究,将其与20世纪90年代的TREC ad hoc排名案例进行比较。我们展示了如何设计的评价工作可以鼓励或阻止某些结果,并提出了内部和外部的有效性的结果的问题。我们对某些陷阱进行了一些分析,并陈述了避免此类陷阱的最佳实践。我们总结了到目前为止的工作进展,并描述了我们期望的“强大的有用性”的最终状态,以及可能需要采取的步骤。
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University(香港理工大学);University of Electronic Science and Technology of China(电子科技大学);Tencent Youtu Lab(腾讯优图实验室);Beihang University(北京航空航天大学);Simon Fraser University(西蒙弗雷泽大学)
PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language
School of Computer Science Institute for Research in Fundamental Sciences(计算机科学学院基础科学研究研究所);School of Electrical and Computer Engineering University of Tehran(电气与计算机工程学院塔里班大学);Department of Computer Science Faculty of Mathematical Sciences Alzahra University(计算机科学系数学科学学院阿扎赫拉大学)
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
University of Tokyo(东京大学);University College London(伦敦大学学院);Tohoku University(东北大学);RIKEN Center for AIP(理化学研究所人工智能中心);National Institute of Informatics(国立信息学研究所)
Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates
University of Alberta(阿尔伯塔大学);Stanford Center for Biomedical Informatics Research(斯坦福大学生物医学信息学研究中心);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Mass General Brigham(麻省总医院布里格姆);Harvard Medical School(哈佛医学院);Stanford University School of Medicine(斯坦福大学医学院);Stanford Mussallem Center for Biodesign(斯坦福穆萨勒姆生物设计中心);Stanford Health Care(斯坦福医疗保健);University of British Columbia(不列颠哥伦比亚大学);Stanford School of Medicine(斯坦福医学院)
UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions
College of Software, Nankai University(南开大学软件学院);Institute for Advanced Algorithms Research (Shanghai)(上海先进算法研究所);College of Artificial Intelligence, Tianjin University of Science and Technology(天津科技大学人工智能学院);Tianjin Key Laboratory of Software Experience and Human Computer Interaction(天津软件体验与人机交互重点实验室)
CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering
Department of Computer Science, University of Southern California(南加州大学计算机科学系);Department of Electrical and Computer Engineering, University of Southern California(南加州大学电气与计算机工程系);Suzanne Dworak-Peck School of Social Work, University of Southern California(南加州大学苏兹安·德沃拉克-佩克社会工作学院);Department of Psychiatry and the Behavioral Sciences, University of Southern California(南加州大学精神病学与行为科学系);Annenberg School for Communication, University of Southern California(南加州大学安纳伯格通信学院)
NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
Westlake University(西湖大学);University of Waterloo(多伦多大学);Hangzhou Normal University(杭州师范大学);Shanghai AI Lab(上海人工智能实验室);SJTU(上海交通大学);NYU Shanghai(纽约大学上海分校)