教育实验中的持续复合效应和敏捷实践
Towards Continuous Compounding Effects and Agile Practices in Educational Experimentation
摘要:
随机对照试验目前是正式教育实验的权威黄金标准方法。虽然这些实验的结论是高度可信的,但其相对缓慢的实验速度、高费用和刚性框架可以被视为限制了以下方面的范围:1. $\textit{metrics}$:自动化每次实验的数百个指标的一致严格计算; 2. $\textit{concurrency}$:每天快速自动发布数百个并发实验;以及3. $\textit{safeguards}$:安全网测试和快速增加/减少治疗,以最大限度地减少负面影响。本文定义了一个框架,分类不同的实验过程,并特别强调技术准备。在我们分析的基础上,我们的论文是,下一代教育技术的成功将预示着认识到实验的背景下,并共同拥抱手头的全套过程:从快速构思和原型制作在小规模实验一方面,影响建议的最佳教学实践与大规模和技术支持的在线A/B测试。后者的一个主要好处是,运行成本趋于零(导致“免费试验”)。这提供了低风险的机会,通过精心策划的持久活动来探索和推动价值,这些活动可以快速大规模地进行。重要的是,由于这些实验平台具有很强的适应性,即使每个单独的实验产生很小的影响,实验活动的累积效应也会随着时间的推移呈指数级地产生复合价值。
英文摘要:
Randomised control trials are currently the definitive gold standard approach for formal educational experiments. Although conclusions from these experiments are highly credible, their relatively slow experimentation rate, high expense and rigid framework can be seen to limit scope on: 1. $\textit{metrics}$: automation of the consistent rigorous computation of hundreds of metrics for every experiment; 2. $\textit{concurrency}$: fast automated releases of hundreds of concurrent experiments daily; and 3. $\textit{safeguards}$: safety net tests and ramping up/rolling back treatments quickly to minimise negative impact. This paper defines a framework for categorising different experimental processes, and places a particular emphasis on technology readiness. On the basis of our analysis, our thesis is that the next generation of education technology successes will be heralded by recognising the context of experiments and collectively embracing the full set of processes that are at hand: from rapid ideation and prototyping produced in small scale experiments on the one hand, to influencing recommendations of best teaching practices with large-scale and technology-enabled online A/B testing on the other. A key benefit of the latter is that the running costs tend towards zero (leading to `free experimentation'). This offers low-risk opportunities to explore and drive value though well-planned lasting campaigns that iterate quickly at a large scale. Importantly, because these experimental platforms are so adaptable, the cumulative effect of the experimental campaign delivers compounding value exponentially over time even if each individual experiment delivers a small effect.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。