教育技术学随机评估的学生日志数据：因果案例研究
Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study
摘要：
教育技术的随机评估产生的日志数据作为一个双产品：高粒度的数据，学生和教师的使用。这些数据集可以揭示因果机制，影响异质性或最佳使用。然而，存在方法上的挑战：实施不是随机的，仅针对治疗组定义，日志数据集具有复杂的结构。本文讨论了三种方法来帮助克服这些问题。一种方法使用来自治疗组的数据来估计使用对观察性研究结果的影响。另一种是因果中介分析，估计使用在推动整体效果中的作用。最后，主要分层估计具有相同的“潜在”使用的学生群体的整体效果。我们使用这三种方法分析了认知导师代数I课程评估的提示数据，结果可能相互矛盾：观察性研究和中介分析表明，提示降低后测分数，而主要分层发现，处理效应可能与较高的提示请求率相关。我们讨论这些混合的结论，并给出更广泛的方法建议。
英文摘要：
Randomized evaluations of educational technology produce log data as a bi-product: highly granular data student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This paper discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
