arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.04429cs.HC
收藏

你真的编码吗?使用程序员设计和评估在线调查的筛选问题

Do you really code? Designing and Evaluating Screening Questions for Online Surveys with Programmers

Anastasia Danilova, Alena Naiakshina, Stefan Horstmann, Matthew Smith

摘要:

招聘足够数量的专业程序员进行研究可能具有挑战性,因为他们往往抽不出时间,或者由于他们的地理分布和潜在的成本。Clickworker或Qualtrics等在线平台确实提供了招募具有编程技能的参与者的选项;然而,误解和欺诈可能是一个问题。这可能导致没有编程技能的参与者参加研究和调查。如果这些参与者没有被检测到,他们可能会在调查数据中造成有害的噪音。在本文中,我们开发了筛选问题,这些问题对于具有编程技能的人来说很容易快速回答,但对于没有编程技能的人来说很难正确回答。为了评估我们的问卷的有效性和效率,我们招募了几批具有和不具有编程技能的参与者,并测试了这些问题。在我们的批次中,42%的点击工人表示他们有编程技能不符合我们的标准,我们建议从研究中过滤这些。我们还在对抗性环境中评估了这些问题。最后,我们提出了一组建议的问题,研究人员可以使用这些问题从在线平台上招募具有编程技能的参与者。

英文摘要:

Recruiting professional programmers in sufficient numbers for research studies can be challenging because they often cannot spare the time, or due to their geographical distribution and potentially the cost involved. Online platforms such as Clickworker or Qualtrics do provide options to recruit participants with programming skill; however, misunderstandings and fraud can be an issue. This can result in participants without programming skill taking part in studies and surveys. If these participants are not detected, they can cause detrimental noise in the survey data. In this paper, we develop screener questions that are easy and quick to answer for people with programming skill but difficult to answer correctly for those without. In order to evaluate our questionnaire for efficacy and efficiency, we recruited several batches of participants with and without programming skill and tested the questions. In our batch 42% of Clickworkers stating that they have programming skill did not meet our criteria and we would recommend filtering these from studies. We also evaluated the questions in an adversarial setting. We conclude with a set of recommended questions which researchers can use to recruit participants with programming skill from online platforms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑