ISD-Agent-Bench: 一个用于评估基于大语言模型的教学系统设计代理的全面基准
ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
AI总结：
ISD-Agent-Bench通过结合经典ISD理论与现代推理方法，为评估基于LLM的教学系统设计代理提供了全面的基准。
摘要：
大型语言模型（LLM）代理在自动化教学系统设计（ISD）方面显示出巨大的潜力，这是一种系统化的方法来开发教育程序。然而，评估这些代理仍然具有挑战性，因为缺乏标准化的基准，并且存在LLM作为评判者偏差的风险。我们提出了ISD-Agent-Bench，一个包含25,795个场景的全面基准，这些场景通过上下文矩阵框架生成，该框架结合了51个上下文变量（跨5个类别）和33个来自ADDIE模型的ISD子步骤。为了确保评估的可靠性，我们采用多评判协议，使用不同供应商的多样化LLM，实现了高评判者可靠性。我们比较了现有的ISD代理与基于经典ISD理论（如ADDIE、Dick与Carey、快速原型ISD）的新代理。在1,017个测试场景上的实验表明，将经典ISD框架与现代ReAct风格推理相结合，能够实现最高性能，优于纯理论基础代理和技术-only方法。进一步分析发现，理论质量与基准性能之间存在强相关性，理论基础代理在问题导向设计和目标评估对齐方面显示出显著优势。我们的工作为系统化的基于LLM的ISD研究提供了基础。
英文摘要：
Large Language Model (LLM) agents have shown promising potential in automating Instructional Systems Design (ISD), a systematic approach to developing educational programs. However, evaluating these agents remains challenging due to the lack of standardized benchmarks and the risk of LLM-as-judge bias. We present ISD-Agent-Bench, a comprehensive benchmark comprising 25,795 scenarios generated via a Context Matrix framework that combines 51 contextual variables across 5 categories with 33 ISD sub-steps derived from the ADDIE model. To ensure evaluation reliability, we employ a multi-judge protocol using diverse LLMs from different providers, achieving high inter-judge reliability. We compare existing ISD agents with novel agents grounded in classical ISD theories such as ADDIE, Dick \& Carey, and Rapid Prototyping ISD. Experiments on 1,017 test scenarios demonstrate that integrating classical ISD frameworks with modern ReAct-style reasoning achieves the highest performance, outperforming both pure theory-based agents and technique-only approaches. Further analysis reveals that theoretical quality strongly correlates with benchmark performance, with theory-based agents showing significant advantages in problem-centered design and objective-assessment alignment. Our work provides a foundation for systematic LLM-based ISD research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
