arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.06498cs.LG
收藏

两个世界中最糟糕的:心理学和机器学习中数据学习错误的比较分析

The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning

Northwestern University(西北大学);Princeton University(普林斯顿大学);Columbia University(哥伦比亚大学);Microsoft Research(微软研究院)

Jessica Hullman, Sayash Kapoor, Priyanka Nanayakkara, Andrew Gelman, Arvind Narayanan

摘要:

最近关于机器学习(ML)正面临再现性和复制危机的争论表明,ML研究中的一些已发表的声明不能从表面上看。这些担忧激发了对影响社会和医学科学的复制危机的类比。他们还激发了对因果推理和预测建模的统计方法的整合的呼吁。更深入地了解监督ML研究中的再现性问题与实验科学中的复制危机的共同点,可以正确看待新的问题,并帮助研究人员避免“两个世界中最糟糕的”,即ML研究人员开始从解释性建模中借用方法,而不了解它们的局限性,反之亦然。我们贡献了一个比较分析的关注归纳学习,出现在因果归因的心理学与预测建模,以ML为例。我们确定了在改革讨论中重新出现的主题,如过度依赖渐近理论和对真实世界数据生成过程的不可信信念。我们认为,在这两个领域中,来自学习的主张都隐含在所研究的特定环境之外(例如,输入数据集或主题样本、建模实现等)但是由于学习管道中未公开的方差源而常常不可能反驳。特别是,ML中承认的错误暴露了长期以来的信念,即使用巨大的数据集优化预测准确性可以免除人们必须考虑真实的数据生成过程或正式表示性能声明中的不确定性。最后,我们讨论了错误来源被误诊时产生的风险,以及承认人类归纳偏见在学习和改革中的作用的必要性。

英文摘要:

Recent arguments that machine learning (ML) is facing a reproducibility and replication crisis suggest that some published claims in ML research cannot be taken at face value. These concerns inspire analogies to the replication crisis affecting the social and medical sciences. They also inspire calls for the integration of statistical approaches to causal inference and predictive modeling. A deeper understanding of what reproducibility concerns in supervised ML research have in common with the replication crisis in experimental science puts the new concerns in perspective, and helps researchers avoid "the worst of both worlds," where ML researchers begin borrowing methodologies from explanatory modeling without understanding their limitations and vice versa. We contribute a comparative analysis of concerns about inductive learning that arise in causal attribution as exemplified in psychology versus predictive modeling as exemplified in ML. We identify themes that re-occur in reform discussions, like overreliance on asymptotic theory and non-credible beliefs about real-world data generating processes. We argue that in both fields, claims from learning are implied to generalize outside the specific environment studied (e.g., the input dataset or subject sample, modeling implementation, etc.) but are often impossible to refute due to undisclosed sources of variance in the learning pipeline. In particular, errors being acknowledged in ML expose cracks in long-held beliefs that optimizing predictive accuracy using huge datasets absolves one from having to consider a true data generating process or formally represent uncertainty in performance claims. We conclude by discussing risks that arise when sources of errors are misdiagnosed and the need to acknowledge the role of human inductive biases in learning and reform.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑