arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.05310cs.SE
收藏

连续实验中的受控实验:知识与挑战

Controlled Experimentation in Continuous Experimentation: Knowledge and Challenges

Florian Auer, Rasmus Ros, Lukas Kaltenbrunner, Per Runeson, Michael Felderer

摘要:

背景:持续实验和A/B测试是一种成熟的行业实践,已经研究了10多年。我们的目标是综合所进行的研究。目标:我们希望找到持续实验框架的核心组成部分以及在该领域应用的解决方案。最后,我们对持续实验的挑战和好处感兴趣。方法:我们在一组已知的论文上应用前向滚雪球法,共识别出128篇相关论文。基于这组论文,我们进行了两个定性的叙述性综合和一个主题性综合来回答研究问题。结果如下:持续实验的框架组成部分包括实验过程以及支持性的技术和组织基础设施。在文献中找到的解决方案被合成为九个主题,例如,实验设计,自动化实验,或度量规格。关于持续实验的挑战,分析确定了文化,组织,业务,技术,统计,道德和特定领域的挑战。此外,该研究得出结论,实验的好处大多隐含在研究中。结论:对连续实验的研究已经产生了大量的实验知识。其中介绍的已发表研究的综合包括推荐的基础设施和实验过程模型,减轻已确定挑战的指导方针,以及各种已发表解决方案解决的问题。

英文摘要:

Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Method: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusions: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑