调查众包以产生多项选择评估的干扰因素
Investigating Crowdsourcing to Generate Distractors for Multiple-Choice Assessments
摘要:
我们提出并分析了一项试点研究的结果,该研究探讨了如何在多项选择概念清单(理解的概念测试)中产生干扰因素(不正确的答案选择)的过程中使用众包。据我们所知,我们是第一个提出和研究这种方法的。使用Amazon Mechanical Turk,我们收集了来自网络安全评估工具项目的网络安全概念清单和数字逻辑概念清单的几个问题的大约180个开放式回答。我们通过过滤响应、对相似响应进行分组、选择四个最频繁的组并为每个组精炼一个代表性的干扰项来生成初步的干扰项。我们从两个方面分析了数据。首先,我们比较了反应和由此产生的干扰与上述清单。第二,我们从Amazon Mechanical Turk获得了关于其他受试者的新测试项目草案(包括干扰项)的反馈。使用众包的挑战包括控制主题的选择和过滤掉不反映真正努力的答复。尽管有这些挑战,我们的研究结果表明,众包可以是一个非常有用的工具,在产生有效的干扰(有吸引力的主题谁不理解的目标概念)。我们的研究结果还表明,这种方法比传统的方法更快,更容易,更便宜,传统的方法是让一个或多个专家起草干扰物,并与受试者进行大声交谈,以发现他们的误解。我们的结果是重要的,因为产生有效的干扰是创建多项选择评估中最困难的步骤之一。
英文摘要:
We present and analyze results from a pilot study that explores how crowdsourcing can be used in the process of generating distractors (incorrect answer choices) in multiple-choice concept inventories (conceptual tests of understanding). To our knowledge, we are the first to propose and study this approach. Using Amazon Mechanical Turk, we collected approximately 180 open-ended responses to several question stems from the Cybersecurity Concept Inventory of the Cybersecurity Assessment Tools Project and from the Digital Logic Concept Inventory. We generated preliminary distractors by filtering responses, grouping similar responses, selecting the four most frequent groups, and refining a representative distractor for each of these groups. We analyzed our data in two ways. First, we compared the responses and resulting distractors with those from the aforementioned inventories. Second, we obtained feedback from Amazon Mechanical Turk on the resulting new draft test items (including distractors) from additional subjects. Challenges in using crowdsourcing include controlling the selection of subjects and filtering out responses that do not reflect genuine effort. Despite these challenges, our results suggest that crowdsourcing can be a very useful tool in generating effective distractors (attractive to subjects who do not understand the targeted concept). Our results also suggest that this method is faster, easier, and cheaper than is the traditional method of having one or more experts draft distractors, and building on talk-aloud interviews with subjects to uncover their misconceptions. Our results are significant because generating effective distractors is one of the most difficult steps in creating multiple-choice assessments.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。