arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01925cs.SE
收藏

你是一位真正的软件工程师吗?软件工程研究在线招募的最佳实践

Are You a Real Software Engineer? Best Practices in Online Recruitment for Software Engineering Studies

Adam Alami, Mansooreh Zahedi, Neil Ernst

AI总结:

针对在线平台招募软件工程师的挑战,本文基于Prolific的三项研究经验,提出了一套包括迭代预筛选、任务导向问题、AI检测、人工评估、额外筛选和合理补偿的招募最佳实践,以提升SE研究样本的质量与可靠性。

摘要:

在线研究平台(如Prolific)能够快速获取多样化的参与者群体,但也给参与者资格认定和技能验证带来了独特的挑战。以往的研究报告了在利用在线平台招募合格软件工程师方面的混合结果和挑战。基于我们在使用Prolific进行的三项不同研究的经验,我们提出了招募和筛选参与者的最佳实践,以提高软件工程(SE)定性和定量研究样本的质量与相关性。我们提出了在Prolific上进行SE研究招募的精细化最佳实践:(1)迭代且受控的预筛选,能够对提交内容进行聚焦且可管理的评估;(2)面向任务且有针对性的问题,用于评估技术技能、基础SE概念知识以及专业参与度;(3)使用AI检测来验证自由文本回复的真实性;(4)对回复进行定性和人工评估,确保参与者回答的真实性和相关性;(5)在必要时增加额外的预筛选层,以收集与研究主题相关的数据;(6)在资格认定后提供公平或慷慨的补偿,以激励真实参与。通过分享我们的经验和教训,我们为开发有效且严谨的SE实证研究方法做出了贡献,特别是为建立确保可靠数据收集的指南而持续进行的努力。这些实践具有可迁移到其他参与者招募平台的潜力。

英文摘要:

Online research platforms, such as Prolific, offer rapid access to diverse participant pools but also pose unique challenges in participant qualification and skill verification. Previous studies reported mixed outcomes and challenges in leveraging online platforms for the recruitment of qualified software engineers. Drawing from our experience in conducting three different studies using Prolific, we propose best practices for recruiting and screening participants to enhance the quality and relevance of both qualitative and quantitative software engineering (SE) research samples. We propose refined best practices for recruitment in SE research on Prolific. (1) Iterative and controlled prescreening, enabling focused and manageable assessment of submissions (2) task-oriented and targeted questions that assess technical skills, knowledge of basic SE concepts, and professional engagement. (3) AI detection to verify the authenticity of free-text responses. (4) Qualitative and manual assessment of responses, ensuring authenticity and relevance in participant answers (5) Additional layers of prescreening are necessary when necessary to collect data relevant to the topic of the study. (6) Fair or generous compensation post-qualification to incentivize genuine participation. By sharing our experiences and lessons learned, we contribute to the development of effective and rigorous methods for SE empirical research. particularly the ongoing effort to establish guidelines to ensure reliable data collection. These practices have the potential to transferability to other participant recruitment platforms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑