CPG-EVAL：评估大型语言模型中文教学语法能力的多层基准
CPG-EVAL: A Multi-Tiered Benchmark for Evaluating the Chinese Pedagogical Grammar Competence of Large Language Models
AI总结：
本文针对LLMs中文教学语法能力评估不足的问题，提出首个多层基准CPG-EVAL，通过五个任务评估相关能力，揭示了不同规模模型的表现差异，为教育场景LLM部署提供指导并奠定后续研究基础。
摘要：
目的：ChatGPT等大型语言模型（LLMs）的快速涌现对二语教育产生了显著影响，但其教学语法能力的评估仍不充分。本文介绍CPG-EVAL，这是首个专门设计用于评估LLMs在二语教学语境下教学语法知识的基准。方法：该基准包含五个任务，旨在评估语法识别、细粒度语法区分、类别判别及抗语言干扰能力。发现：小规模模型可在单语言实例任务中取得成功，但在多实例任务和混淆实例干扰下表现不佳；大规模模型抗干扰能力更强，但准确率仍有较大提升空间。评估表明需更好的教学对齐和更严格的基准，以有效指导LLMs在教育场景中的部署。价值：本研究提供了首个针对中文教学语境下LLMs教学语法能力的专门、理论驱动的多层基准框架。CPG-EVAL不仅为教育工作者、政策制定者和模型开发者提供了实证见解，以更好地衡量AI在教育场景中的当前能力，还为未来研究（如改进模型对齐、增强教育适用性、确保LLM融入二语教学的知情决策）奠定了基础。
英文摘要：
Purpose: The rapid emergence of large language models (LLMs) such as ChatGPT has significantly impacted foreign language education, yet their pedagogical grammar competence remains under-assessed. This paper introduces CPG-EVAL, the first dedicated benchmark specifically designed to evaluate LLMs' knowledge of pedagogical grammar within the context of foreign language instruction. Methodology: The benchmark comprises five tasks designed to assess grammar recognition, fine-grained grammatical distinction, categorical discrimination, and resistance to linguistic interference. Findings: Smaller-scale models can succeed in single language instance tasks, but struggle with multiple instance tasks and interference from confusing instances. Larger-scale models show better resistance to interference but still have significant room for accuracy improvement. The evaluation indicates the need for better instructional alignment and more rigorous benchmarks, to effectively guide the deployment of LLMs in educational contexts. Value: This study offers the first specialized, theory-driven, multi-tiered benchmark framework for systematically evaluating LLMs' pedagogical grammar competence in Chinese language teaching contexts. CPG-EVAL not only provides empirical insights for educators, policymakers, and model developers to better gauge AI's current abilities in educational settings, but also lays the groundwork for future research on improving model alignment, enhancing educational suitability, and ensuring informed decision-making concerning LLM integration in foreign language instruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
