Shaky Structures:软件分析中因果图的不稳定世界
Shaky Structures: The Wobbly World of Causal Graphs in Software Analytics
AI总结:
该研究揭示软件工程中因果图生成结果高度不稳定的问题,通过多生成器、多数据集、多维度对比验证其变异性,提出相关结论需经多图一致性检验的建议。
摘要:
因果图在软件工程领域被广泛用于记录和探究因果关系。尽管应用普遍,但它们也可能极具误导性。从软件工程数据生成的因果结构可能具有高度可变性。这种不稳定性极为显著,以至于从一张图中得出的结论在另一张图中可能完全相反,即便两张图都是从相同或高度相似的项目数据中学习得到的。\n 为了记录这一问题,本文研究了四种因果图生成器(PC、FCI、GES 和 LiNGAM)应用于23个数据集时得到的因果图,这些数据集涉及三项不同的软件工程任务:(a) 研究配置选项如何针对不同属性被选择;(b) 理解管理选择如何影响软件项目;(c) 缺陷预测。我们从以下维度对比了因果图:(a) 探索同一任务的不同项目之间;(b) 系统的第i版与第i+1版之间;(c) 数据的不同90%样本之间;(d) 因果图生成器的微小变动带来的差异。以两张不同图共享边数的Jaccard指数衡量,这些处理导致超过一半的边发生了变化。\n 因此,我们得出两点结论。第一,因果图生成器得出的关于两个特定变量如何相互影响的具体结论可能不具备普适性,因为这些图生成方式的微小变动就可能推翻这些结论。第二,研究人员在报告据称可从因果图得出的普适性结论(例如“长函数会导致更多缺陷”)之前,应当检验这些结论在从同一数据可能生成的众多因果图中是否都成立。
英文摘要:
Causal graphs are widely used in software engineering to document and explore causal relationships. Though widely used, they may also be wildly misleading. Causal structures generated from SE data can be highly variable. This instability is so significant that conclusions drawn from one graph may be totally reversed in another, even when both graphs are learned from the same or very similar project data. To document this problem, this paper examines causal graphs found by four causal graph generators (PC, FCI, GES, and LiNGAM) when applied to 23 data sets, relating to three different SE tasks: (a) learning how configuration options are selected for different properties; (b) understanding how management choices affect software projects; and (c) defect prediction. Graphs were compared between (a) different projects exploring the same task; (b) version i and i + 1 of a system; (c) different 90% samples of the data; and (d) small variations in the causal graph generator. Measured in terms of the Jaccard index of the number of edges shared by two different graphs, over half the edges were changed by these treatments. Hence, we conclude two things. Firstly, specific conclusions found by causal graph generators about how two specific variables affect each other may not generalize since those conclusions could be reversed by minor changes in how those graphs are generated. Secondly, before researchers can report supposedly general conclusions from causal graphs (e.g., "long functions cause more defects"), they should test that such conclusions hold over the numerous causal graphs that might be generated from the same data.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。