CodeGuard: 提高计算机科学教育中的LLM防护机制
CodeGuard: Improving LLM Guardrails in CS Education
AI总结：
CodeGuard通过引入新的分类法、数据集和PromptShield模型，有效提升教育AI系统对不安全提示的检测能力，减少有害代码生成，同时保持教育任务性能。
摘要：
大型语言模型（LLMs）越来越多地嵌入到计算机科学（CS）课堂中，以自动化代码生成、反馈和评估。然而，它们对对抗性或恶意提示的易受攻击性威胁学生学习和学术诚信。为应对这一重要问题，我们评估了现有现成的LLMs在处理CS教育领域中不安全和无关提示的能力。我们发现现有LLM防护机制的重要不足，这促使我们提出CodeGuard，一种全面的教育AI系统防护框架。CodeGuard包括（i）一种首次出现的分类法，用于分类提示；（ii）CodeGuard数据集，一个包含8000个提示的集合，覆盖该分类法；以及（iii）PromptShield，一个轻量级的句子编码器模型，经过微调以实时检测不安全提示。实验显示，PromptShield实现了0.93的F1得分，超过了现有防护方法。此外，进一步实验表明，CodeGuard减少了潜在有害或违反政策的代码生成，减少了30-65%，而不会影响教育任务的性能。代码、数据集和评估脚本已免费向社区提供。
英文摘要：
Large language models (LLMs) are increasingly embedded in Computer Science (CS) classrooms to automate code generation, feedback, and assessment. However, their susceptibility to adversarial or ill-intentioned prompts threatens student learning and academic integrity. To cope with this important issue, we evaluate existing off-the-shelf LLMs in handling unsafe and irrelevant prompts within the domain of CS education. We identify important shortcomings in existing LLM guardrails which motivates us to propose CodeGuard, a comprehensive guardrail framework for educational AI systems. CodeGuard includes (i) a first-of-its-kind taxonomy for classifying prompts; (ii) the CodeGuard dataset, a collection of 8,000 prompts spanning the taxonomy; and (iii) PromptShield, a lightweight sentence-encoder model fine-tuned to detect unsafe prompts in real time. Experiments show that PromptShield achieves 0.93 F1 score, surpassing existing guardrail methods. Additionally, further experimentation reveals that CodeGuard reduces potentially harmful or policy-violating code completions by 30-65% without degrading performance on legitimate educational tasks. The code, datasets, and evaluation scripts are made freely available to the community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
