arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.10563cs.SE
收藏

程序理解实验中有效性威胁的证据画像

Evidence Profiles for Validity Threats in Program Comprehension Experiments

Marvin Muñoz Barón, Marvin Wyrich, Daniel Graziotin, Stefan Wagner

AI总结:

通过系统文献检索与主题综合,构建程序理解实验中编程经验、程序长度和理解测量三类最常见有效性威胁的证据画像,揭示现有威胁讨论普遍缺乏证据支持,并倡导开展更多基于证据的元研究以改进实验设计。

摘要:

寻找线索、收集证据和审查案卷都是刑事调查人员用来得出可靠结论并避免错误定罪的技术。类似地,在软件工程(SE)研究中,我们可以通过将研究设计决策建立在证据之上,来发展可靠的方法论并减轻对有效性的威胁。\n响应近期关于对程序理解实验中的设计决策进行实证评估的呼吁,我们开展了一项两阶段研究,包括系统文献检索、滚雪球式检索和主题综合。我们发现了(1)在代码理解的主要研究中,哪些有效性威胁类别最常被讨论,并且我们收集了证据来构建(2)三种最常报告的有效性威胁的证据画像。\n我们发现,主要研究中很少提及有效性威胁(409篇中有31篇)包含对支持性证据的引用。对于三种最常被提及的威胁,即编程经验的影响、程序长度以及所选的程序理解测量指标,几乎所有被引用的研究(18篇中有17篇)都不符合我们的证据标准。我们表明,对于许多目前被认为在所有研究中都具有影响力的有效性威胁,其实际影响可能取决于每项具体研究的设计和情境。\n研究人员应在其特定研究的情境中讨论有效性威胁,并用证据支持他们的讨论。本文可以作为证据的一种资源,我们呼吁开展更多此类元研究,这些研究随后将为主要研究中的设计决策提供信息。此外,尽管我们已在程序理解的背景下应用了我们的方法,但我们的方法也可以用于其他软件工程研究领域,以实现基于证据的实验设计决策和对有效性威胁的有意义讨论。

英文摘要:

Searching for clues, gathering evidence, and reviewing case files are all techniques used by criminal investigators to draw sound conclusions and avoid wrongful convictions. Similarly, in software engineering (SE) research, we can develop sound methodologies and mitigate threats to validity by basing study design decisions on evidence. Echoing a recent call for the empirical evaluation of design decisions in program comprehension experiments, we conducted a 2-phases study consisting of systematic literature searches, snowballing, and thematic synthesis. We found out (1) which validity threat categories are most often discussed in primary studies of code comprehension, and we collected evidence to build (2) the evidence profiles for the three most commonly reported threats to validity. We discovered that few mentions of validity threats in primary studies (31 of 409) included a reference to supporting evidence. For the three most commonly mentioned threats, namely the influence of programming experience, program length, and the selected comprehension measures, almost all cited studies (17 of 18) did not meet our criteria for evidence. We show that for many threats to validity that are currently assumed to be influential across all studies, their actual impact may depend on the design and context of each specific study. Researchers should discuss threats to validity within the context of their particular study and support their discussions with evidence. The present paper can be one resource for evidence, and we call for more meta-studies of this type to be conducted, which will then inform design decisions in primary studies. Further, although we have applied our methodology in the context of program comprehension, our approach can also be used in other SE research areas to enable evidence-based experiment design decisions and meaningful discussions of threats to validity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑