创建和验证网络安全概念清单的经验和教训
Experiences and Lessons Learned Creating and Validating Concept Inventories for Cybersecurity
摘要：
我们反思了我们在教育网络安全评估工具（CATS）项目中正在进行的旅程，以创建两个网络安全概念清单。我们确定了这一旅程中的关键步骤和我们面临的重要问题。我们会解释我们所做的决定，并讨论这些决定的后果，强调哪些是有效的，哪些可能会更好。CATS项目正在创建和验证两个概念清单-理解的概念测试-可用于衡量各种网络安全教学方法的有效性。网络安全概念清单（CCI）适用于最近完成网络安全第一门课程的学生;网络安全课程评估（CCA）适用于最近完成网络安全本科专业或课程的学生。每个评估工具包括25个不同难度的多项选择题，针对相同的五个核心概念，但共同国家评估假设更多的技术背景。关键步骤包括定义项目范围，确定核心概念，发现学生的误解，创建场景，起草问题干，开发分心的答案选择，生成教育材料，执行专家评审，招募学生科目，组织研讨会，建立社区接受度，组建团队并培养合作，采用工具，以及获得和使用资金。创建有效的MCQ既困难又耗时，网络安全也带来了特殊的挑战。由于网络安全问题往往是微妙的，其中对抗模型和细节非常重要，因此构建MCQ具有挑战性，因为只有一个最佳但不明显的答案。我们希望我们的经验和教训可以帮助其他人在STEM中创建更有效的概念库存和评估。
英文摘要：
We reflect on our ongoing journey in the educational Cybersecurity Assessment Tools (CATS) Project to create two concept inventories for cybersecurity. We identify key steps in this journey and important questions we faced. We explain the decisions we made and discuss the consequences of those decisions, highlighting what worked well and what might have gone better. The CATS Project is creating and validating two concept inventories---conceptual tests of understanding---that can be used to measure the effectiveness of various approaches to teaching and learning cybersecurity. The Cybersecurity Concept Inventory (CCI) is for students who have recently completed any first course in cybersecurity; the Cybersecurity Curriculum Assessment (CCA) is for students who have recently completed an undergraduate major or track in cybersecurity. Each assessment tool comprises 25 multiple-choice questions (MCQs) of various difficulties that target the same five core concepts, but the CCA assumes greater technical background. Key steps include defining project scope, identifying the core concepts, uncovering student misconceptions, creating scenarios, drafting question stems, developing distractor answer choices, generating educational materials, performing expert reviews, recruiting student subjects, organizing workshops, building community acceptance, forming a team and nurturing collaboration, adopting tools, and obtaining and using funding. Creating effective MCQs is difficult and time-consuming, and cybersecurity presents special challenges. Because cybersecurity issues are often subtle, where the adversarial model and details matter greatly, it is challenging to construct MCQs for which there is exactly one best but non-obvious answer. We hope that our experiences and lessons learned may help others create more effective concept inventories and assessments in STEM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
