MIRROR：一种用于开放式问题生成自动评估的新方法
MIRROR: A Novel Approach for the Automated Evaluation of Open-Ended Question Generation
AI总结：
本文提出MIRROR系统，利用大型语言模型自动评估开放式问题生成质量，通过反馈迭代提升评分与人类评估一致性，显著改善相关性和适当性。
摘要：
自动问题生成是一项关键任务，涉及通过考虑参与度、教学价值以及激发批判性思维的能力等因素来评估问题质量。这些方面需要类人的理解和判断，而当前的自动化系统尚不具备这种能力。然而，人工评估成本高昂，且对于大规模生成的问题样本来说不切实际。因此，我们提出了一种新系统MIRROR（多大型语言模型迭代审查与响应以实现优化评分），该系统利用大型语言模型（LLMs）来自动化评估由自动问题生成系统生成的问题。我们实验了几种最先进的大型语言模型，如GPT-4、Gemini和Llama2-70b。我们观察到，当使用基于反馈的方法MIRROR时，人工评估指标（即相关性、适当性、新颖性、复杂性和语法性）的得分有所提高，趋向于更接近人工基线得分。此外，我们观察到，与直接提示评估相比，使用我们提出的基于反馈的方法MIRROR时，GPT-4与人类专家之间的皮尔逊相关系数有所提高。错误分析表明，我们提出的方法MIRROR显著有助于提高相关性和适当性。
英文摘要：
Automatic question generation is a critical task that involves evaluating question quality by considering factors such as engagement, pedagogical value, and the ability to stimulate critical thinking. These aspects require human-like understanding and judgment, which automated systems currently lack. However, human evaluations are costly and impractical for large-scale samples of generated questions. Therefore, we propose a novel system, MIRROR (Multi-LLM Iterative Review and Response for Optimized Rating), which leverages large language models (LLMs) to automate the evaluation process for questions generated by automated question generation systems. We experimented with several state-of-the-art LLMs, such as GPT-4, Gemini, and Llama2-70b. We observed that the scores of human evaluation metrics, namely relevance, appropriateness, novelty, complexity, and grammaticality, improved when using the feedback-based approach called MIRROR, tending to be closer to the human baseline scores. Furthermore, we observed that Pearson's correlation coefficient between GPT-4 and human experts improved when using our proposed feedback-based approach, MIRROR, compared to direct prompting for evaluation. Error analysis shows that our proposed approach, MIRROR, significantly helps to improve relevance and appropriateness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
