arXiv 2410.131912025-02-10cs.CLcs.AI

MCQG-SRefine：基于迭代自我批判、修正和比较反馈的多项选择题生成与评估

MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback

University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；University of Minnesota（明尼苏达大学）；University of Massachusetts, Lowell（马萨诸塞大学洛厄尔分校）；UMass Chan Medical School（马萨诸塞大学陈医学院）

Zonghai Yao, Aditya Parashar, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, Zhichao Yang, Hong Yu

备注：: Equal contribution for the first two authors. To appear in proceedings of the Main Conference on 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL). Keywords: Question Generation, USMLE, Self-Refine, Self-Critique, and Self-Correction, LLM-as-Judge, AI for Medical Education

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对GPT-4等LLM在专业多项选择题生成（如USMLE题目）中存在的知识过时、幻觉等问题，提出MCQG-SRefine框架，通过迭代自我批判修正及专家提示工程提升题目质量，并引入LLM-as-Judge自动评估指标替代专家评估。

摘要：

自动问题生成（QG）对AI和NLP至关重要，尤其在智能辅导、对话系统和事实核查领域。为专业考试（如美国医师执照考试USMLE）生成多项选择题（MCQG）极具挑战，需要领域专业知识和复杂的多跳推理才能产出高质量题目。然而，GPT-4等现有大语言模型（LLMs）因知识过时、幻觉问题和提示词敏感性，在专业MCQG任务中表现不佳，导致题目质量和难度难以满足需求。为此，我们提出MCQG-SRefine——一种基于LLM自我优化（批判与修正）的框架，用于将医疗案例转化为USMLE风格的高质量题目。通过整合专家驱动的提示词工程与迭代自我批判、自我修正反馈，MCQG-SRefine显著提升了人类专家对题目质量和难度的满意度。此外，我们引入基于LLM-as-Judge的自动评估指标，替代复杂昂贵的专家评估流程，确保评估结果可靠且与专家判断一致。

英文摘要：

Automatic question generation (QG) is essential for AI and NLP, particularly in intelligent tutoring, dialogue systems, and fact verification. Generating multiple-choice questions (MCQG) for professional exams, like the United States Medical Licensing Examination (USMLE), is particularly challenging, requiring domain expertise and complex multi-hop reasoning for high-quality questions. However, current large language models (LLMs) like GPT-4 struggle with professional MCQG due to outdated knowledge, hallucination issues, and prompt sensitivity, resulting in unsatisfactory quality and difficulty. To address these challenges, we propose MCQG-SRefine, an LLM self-refine-based (Critique and Correction) framework for converting medical cases into high-quality USMLE-style questions. By integrating expert-driven prompt engineering with iterative self-critique and self-correction feedback, MCQG-SRefine significantly enhances human expert satisfaction regarding both the quality and difficulty of the questions. Furthermore, we introduce an LLM-as-Judge-based automatic metric to replace the complex and costly expert evaluation process, ensuring reliable and expert-aligned assessments.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。