用于教育应用的LLM提示评估
LLM Prompt Evaluation for Educational Applications
AI总结：
本研究提出了一种系统评估方法，通过测试六个提示模板，发现一个结合角色和上下文管理器模式的提示在教育应用中表现最佳，支持元认知学习策略。
摘要：
随着大型语言模型（LLMs）在教育应用中日益普及，有必要建立基于证据的方法来设计和评估生成个性化且符合教学目标的LLM提示。本研究提出了一种通用且系统的评估方法，通过分析结构对话活动中生成的LLM后续问题进行演示。设计并测试了六个提示模板。这些模板结合了已建立的提示工程模式，每个提示强调不同的教学策略。通过一种竞赛式评估框架进行比较，该框架可适应其他教育应用。竞赛采用Glicko2评分系统，八名评委在三个维度上评估问题对：格式、对话支持和对学习者的适切性。数据来源于120个真实用户交互，涵盖三个不同的教育部署。结果显示，一个与战略阅读相关的提示在 pairwise 比较中胜率范围从81%到100%，该提示结合了角色和上下文管理器模式，旨在支持元认知学习策略，如自主学习。该方法展示了教育技术研究者如何系统地评估和改进提示设计，从随意的提示工程迈向基于证据的提示开发。
英文摘要：
As large language models (LLMs) become increasingly common in educational applications, there is a growing need for evidence-based methods to design and evaluate LLM prompts that produce personalized and pedagogically aligned out-puts. This study presents a generalizable, systematic approach for evaluating prompts, demonstrated through an analysis of LLM-generated follow-up questions in a structured dialogue activity. Six prompt templates were designed and tested. The templates incorporated established prompt engineering patterns, with each prompt emphasizing distinct pedagogical strategies. The prompt templates were compared through a tournament-style evaluation framework that can be adapted for other educational applications. The tournament employed the Glicko2 rating system with eight judges evaluating question pairs across three dimensions: format, dialogue support, and appropriateness for learners. Data was sourced from 120 authentic user interactions across three distinct educational deployments. Results showed that a single prompt related to strategic reading out-performed other templates with win probabilities ranging from 81% to 100% in pairwise comparisons. This prompt combined persona and context manager pat-terns and was designed to support metacognitive learning strategies such as self-directed learning. The methodology showcases how educational technology re- searchers can systematically evaluate and improve prompt designs, moving beyond ad-hoc prompt engineering toward evidence-based prompt development for educational applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
