复制市场：AI复制的结果、教训、挑战和机遇
Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication
摘要：
在过去的十年中，社会和行为科学中出现了系统的大规模复制项目（Camerer等人，2016，2018; Ebersole等人，2016; Klein等人，2014，2018;协作，2015）。这些项目是由对科学出版物中高比例的“假阳性”（Ioannovich，2005）以及“可疑研究实践”（Simmons，Nelson和Simonsohn，2011）的高流行率的理论和概念担忧驱动的。对研究结果可信度的担忧并不是行为科学和社会科学所独有的;在计算机科学中，人工智能（AI）和机器学习（ML）是特别值得关注的领域（Lucic et al.，2018; Freire，Bonnet和Shasha，2012; Gundersen和Kjensmo，2018; Henderson等人，2018年）。鉴于行为和社会科学在促进新方法以提高研究可信度方面的先锋作用，分析从这一领域吸取的经验教训并调整计算机科学，AI和ML的策略是一种很有前途的方法。本文中，我们回顾了行为和社会科学以及DARPA SCORE项目中使用的方法。我们特别关注人类预测复制结果的作用，以及预测如何利用从相对劳动和资源密集型复制中获得的信息。我们将讨论使用这些方法来监控和提高计算机科学，AI和ML研究领域的可信度的机会和挑战。
英文摘要：
The last decade saw the emergence of systematic large-scale replication projects in the social and behavioral sciences, (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2014, 2018; Collaboration, 2015). These projects were driven by theoretical and conceptual concerns about a high fraction of "false positives" in the scientific publications (Ioannidis, 2005) (and a high prevalence of "questionable research practices" (Simmons, Nelson, and Simonsohn, 2011). Concerns about the credibility of research findings are not unique to the behavioral and social sciences; within Computer Science, Artificial Intelligence (AI) and Machine Learning (ML) are areas of particular concern (Lucic et al., 2018; Freire, Bonnet, and Shasha, 2012; Gundersen and Kjensmo, 2018; Henderson et al., 2018). Given the pioneering role of the behavioral and social sciences in the promotion of novel methodologies to improve the credibility of research, it is a promising approach to analyze the lessons learned from this field and adjust strategies for Computer Science, AI and ML In this paper, we review approaches used in the behavioral and social sciences and in the DARPA SCORE project. We particularly focus on the role of human forecasting of replication outcomes, and how forecasting can leverage the information gained from relatively labor and resource-intensive replications. We will discuss opportunities and challenges of using these approaches to monitor and improve the credibility of research areas in Computer Science, AI, and ML.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
