在实践中调和预测多重性
Reconciling Predictive Multiplicity in Practice
AI总结:
本文实证分析了Reconcile算法在模型多重性问题上的有效性,将其与现有方案比较并探讨改进,同时首次将该算法扩展至因果推断场景以解决估计器分歧问题。
摘要:
许多机器学习应用预测个体概率,例如一个人患上特定疾病的可能性。由于这些概率是未知的,一个关键问题是如何处理在相同数据集上训练的不同模型对某些个体产生不同预测的情况。模型多重性(MM)现象正是这一问题的例例,即一组可比的模型产生不一致的预测。Roth、Tolbert 和 Weinstein 最近引入了一种调和程序,即 Reconcile 算法,来解决这个问题。给定两个存在分歧的模型,该算法利用它们的分歧来证伪并改进至少其中一个模型。在本文中,我们使用五个广泛使用的公平性数据集对 Reconcile 算法进行了实证分析:COMPAS、Communities and Crime、Adult、Statlog (German Credit Data) 以及 ACS Dataset。我们探讨了 Reconcile 如何融入模型多重性文献,并将其与现有的 MM 解决方案进行比较,展示了其有效性。我们还从理论和实践上讨论了 Reconcile 算法的潜在改进。最后,鉴于不同的竞争性估计器可能再次在特定的因果平均处理效应(CATE)值上产生分歧,我们将 Reconcile 算法扩展至因果推断场景。我们提出了 Reconcile 算法在因果推断中的首次扩展,分析了其理论性质,并进行了实证测试。我们的结果证实了 Reconcile 的实际有效性及其在各个领域的适用性。
英文摘要:
Many machine learning applications predict individual probabilities, such as the likelihood that a person develops a particular illness. Since these probabilities are unknown, a key question is how to address situations in which different models trained on the same dataset produce varying predictions for certain individuals. This issue is exemplified by the model multiplicity (MM) phenomenon, where a set of comparable models yield inconsistent predictions. Roth, Tolbert, and Weinstein recently introduced a reconciliation procedure, the Reconcile algorithm, to address this problem. Given two disagreeing models, the algorithm leverages their disagreement to falsify and improve at least one of the models. In this paper, we empirically analyze the Reconcile algorithm using five widely-used fairness datasets: COMPAS, Communities and Crime, Adult, Statlog (German Credit Data), and the ACS Dataset. We examine how Reconcile fits within the model multiplicity literature and compare it to existing MM solutions, demonstrating its effectiveness. We also discuss potential improvements to the Reconcile algorithm theoretically and practically. Finally, we extend the Reconcile algorithm to the setting of causal inference, given that different competing estimators can again disagree on specific causal average treatment effect (CATE) values. We present the first extension of the Reconcile algorithm in causal inference, analyze its theoretical properties, and conduct empirical tests. Our results confirm the practical effectiveness of Reconcile and its applicability across various domains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。