arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.00517stat.AP
收藏

预测可复制性--来自大型预测项目的调查预测市场数据分析

Predicting replicability -- analysis of survey and prediction market data from large-scale forecasting projects

Michael Gordon, Domenico Viganola, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer

摘要:

已发表研究的可重复性已成为科学政策中的一个重要议题。已经进行了一些大规模的复制项目,以衡量特定学术领域的整体可重复性。在这里,我们对四项研究的数据进行了分析,这些研究试图预测社会和行为科学中复制项目的结果,使用参与预测市场并回答调查的人类专家。由于在每项单独的研究中重复和预测的结果数量很少,因此汇集数据提供了一个机会,可以在更高的功率下评估关于预测市场和调查表现的假设。总的来说,同行的信念引起了103个已发表的研究结果的复制结果。我们发现,科学界内部有关于科学发现可复制性的信息,调查和预测市场都可以用来获取和汇总这些信息。我们的研究结果表明,预测市场可以以73%的准确率(n=103)确定直接复制的结果。预测市场价格和平均调查响应都与结果相关(分别为0.581和0.564,p <0.001)。我们还发现了原始结果和复制结果的p值之间的显着关系。该数据集通过R包pooledmaRket提供,可用于进一步研究社区对调查和预测市场中得出的复制结果的信念。

英文摘要:

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n=103). Both the prediction market prices and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package pooledmaRket and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑