arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.28482cs.SE
收藏

用结构因果模型重新思考软件实证研究

Rethinking Software Empirical Studies with Structural Causal Models

William & Mary(威廉玛丽学院)

Daniel Rodriguez-Cardenas, Aya Garryyeva, David Nader Palacio, Antonio Mastropaolo, Denys Poshyvanyk

AI总结:

提出CausalSE框架,利用结构因果模型(SCMs)解决混淆偏差,通过Galeras数据集和倾向得分匹配案例,揭示提示工程策略对代码生成的影响,强调因果分析可避免关联分析中的假阳性。

摘要:

因果推断提供了一种基本方法,将经验软件工程(ESE)推进到传统统计关联之外,使研究人员能够严格识别和量化软件实验中的因果关系。本文介绍了CausalSE,这是一个在ESE背景下实现Judea Pearl因果推断范式的框架。论文重点研究结构因果模型(SCMs),以解决经典统计方法在减轻混淆偏差方面的局限性。通过使用Galeras数据集和倾向得分匹配的案例研究,我们展示了CausalSE如何分离提示工程策略对流行LLM(即GPT-3)代码生成结果的影响。结果表明,虽然关联分析可能表明某些干预措施(例如更复杂的提示)有所改进,但因果分析通常未发现显著的处理效应,突出了未处理混淆时假阳性的风险。通过提供基于教程的方法和真实世界的案例研究,这项工作为软件研究人员提供了实用的工具,以严格的方法设计、分析和解释软件实验,最终在研究和实践中实现更明智和可操作的结论。

英文摘要:

Causal Inference offers a fundamental approach for advancing empirical software engineering (ESE) beyond traditional statistical association, enabling researchers to rigorously identify and quantify causal relationships in software experiments. This paper introduces CausalSE, a framework that operationalizes Judea Pearl's causal inference paradigm in ESE context. The paper focuses on Structural Causal Models (SCMs) to address the limitations of classical statistical methods in mitigating confounding bias. Through a case study using the Galeras dataset and propensity score matching, we demonstrate how CausalSE disentangles the effect of prompt engineering strategies on code generation outcomes in a popular LLM (i.e., GPT-3). The results reveal that while associational analyses can suggest improvements in certain interventions (e.g., more complex prompts), causal analysis often does not find a significant treatment effect, highlighting the risk of false positives when confounding is not addressed. By providing a tutorial-based methodology and a real-world case study, this work equips software researchers with practical tools to design, analyze, and interpret software experiments with methodological rigor, ultimately enabling more informed and actionable conclusions in both research and practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑