The automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly. However, existing evaluation metrics for MCQ generation, such as BLEU, ROUGE, and METEOR, focus on the n-gram based similarity of the generated MCQ to the gold sample in the dataset and disregard their educational value. They fail to evaluate the MCQ's ability to assess the student's knowledge of the corresponding target fact. To tackle this issue, we propose a novel automatic evaluation metric, coined Knowledge Dependent Answerability (KDA), which measures the MCQ's answerability given knowledge of the target fact. Specifically, we first show how to measure KDA based on student responses from a human survey. Then, we propose two automatic evaluation metrics, KDA_disc and KDA_cont, that approximate KDA by leveraging pre-trained language models to imitate students' problem-solving behavior. Through our human studies, we show that KDA_disc and KDA_soft have strong correlations with both (1) KDA and (2) usability in an actual classroom setting, labeled by experts. Furthermore, when combined with n-gram based similarity metrics, KDA_disc and KDA_cont are shown to have a strong predictive power for various expert-labeled MCQ quality measures.
Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions
Shanghai University of Finance and Economics(上海财经大学);University of Edinburgh(爱丁堡大学);Southern University of Science and Technology(南方科技大学);Fudan University(复旦大学)
Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answers
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Renmin University of China(中国人民大学)
A Report on the llms evaluating the high school questions
School of Environmental Science and Engineering, Guangdong University of Technology(广东工业大学环境科学与工程学院);School of Computer Science, Guangdong University of Technology(广东工业大学计算机学院)
Can Language Models Analyze Data? Evaluating Large Language Models for Question Answering over Datasets
School of Production Engineering and Management, Technical University of Crete(生产工程与管理学院,希腊克里特技术大学);Institute of Computer Science, Foundation for Research and Technology - Hellas(计算机科学研究所,希腊基础研究与技术研究院)
Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types
School of Computer Science, Harbin University of Science and Technology(哈尔滨理工大学计算机科学学院);Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算学部)
Evaluating Personality Traits in Large Language Models: Insights from Psychological Questionnaires
School of Physics, Mathematics and Computing(物理、数学与计算学院);University of Western Australia(西澳大学);School of Computing(计算机学院);School of Psychological Sciences(心理科学学院)
Hindi Question Generation Using Dependency Structures
Language Technologies Research Centre(语言技术研究中心);Kohli Center on Intelligent Systems(科利智能系统中心);International Institute of Information Technology, Hyderabad(海得拉巴国际信息技术学院)
A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Task
Japan Advanced Institute of Science and Technology(日本先进科学技术学院);National Institute of Advanced Industrial Science and Technology(日本国立先进工业科学技术研究所);OMRON SINIC X Corporation(欧姆龙SINIC X公司)
Knowledge Dependency Estimation for Reliable Question Answering
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);China Industrial Control Systems Cyber Emergency Response Team(中国工业控制系统网络应急响应团队)
AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages
Princeton University(普林斯顿大学);Laboratory for Artificial Intelligence, Princeton University(普林斯顿大学人工智能实验室);Gaston Berger University(加斯顿·伯杰大学);Mila, McGill University(麦吉尔大学米拉研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能教席);Paris Nanterre University(巴黎南泰尔大学);Paris-Saclay University(巴黎-萨克雷大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);LORIA(洛林计算机科学实验室);Université de Lorraine(洛林大学);University of Trento(特伦托大学);University of Minnesota–Twin Cities(明尼苏达大学双城分校);Imperial College London(伦敦帝国学院);Binghamton University(宾汉姆顿大学);Makerere University(马凯雷雷大学);Penn State University(宾夕法尼亚州立大学);Mbarara University of Science and Technology(姆巴拉拉科技大学);Chalmers University of Technology(查尔姆斯理工大学);University of Ibadan(伊巴丹大学);Nnamdi Azikiwe University(纳姆迪·阿齐基韦大学);South African Centre for Digital Language Resources(南非数字语言资源中心)