批判性问题生成基准测试：大型语言模型的一项具有挑战性的推理任务
Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models
AI总结：
针对批判性问题生成（CQs-Gen）缺乏数据集与评估标准的问题，本文构建首个含约5000个人工标注问题的大规模数据集，提出与人类判断相关性最佳的基于参考的自动评估技术，并对11个LLM进行零样本评估建立基线，推动该领域发展。
摘要：
批判性问题生成（CQs-Gen）任务旨在通过让系统生成能揭示潜在假设、挑战论证推理结构有效性的问题来培养批判性思维。尽管该领域关注度不断提升，但由于缺乏合适的数据集和自动评估标准，进展受阻。本文提出一种全面方法以支持该任务系统的开发与基准测试：构建首个包含约5000个人工标注问题的大规模数据集，研究自动评估方法并提出与人类判断相关性最佳的基于参考的技术；对11个LLM进行零样本评估，建立强基线的同时展现任务难度。此外，提供数据、代码及公共排行榜，以鼓励围绕模型性能及CQs-Gen在自动推理和人类批判性思维中的实际效益开展进一步研究。
英文摘要：
The task of Critical Questions Generation (CQs-Gen) aims to foster critical thinking by enabling systems to generate questions that expose underlying assumptions and challenge the validity of argumentative reasoning structures. Despite growing interest in this area, progress has been hindered by the lack of suitable datasets and automatic evaluation standards. This paper presents a comprehensive approach to support the development and benchmarking of systems for this task. We construct the first large-scale dataset including ~5K manually annotated questions. We also investigate automatic evaluation methods and propose reference-based techniques as the strategy that best correlates with human judgments. Our zero-shot evaluation of 11 LLMs establishes a strong baseline while showcasing the difficulty of the task. Data and code plus a public leaderboard are provided to encourage further research, not only in terms of model performance, but also to explore the practical benefits of CQs-Gen for both automated reasoning and human critical thinking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
