arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.03555cs.HC
收藏

非专业人员能否在没有专家干预的情况下预测社会科学研究的可复制性:一项探索性研究

Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study

Juntao Wang, Jonathan Lei, Anna Dreber, Michael Gordon, Magnus Johannesson, Thomas Pfeiffer, Yiling Chen

摘要:

已发表研究的低重复率一直是社会科学界关注的问题,理解可重复性是一个关键问题。几项研究表明,相关的研究团体可以对个别研究的可复制性作出预测,其准确性高于偶然性。后续工作进一步表明,外行人也可以实现上述机会的准确性,在预测可复制性时,专家解释的研究到简短的描述,更容易为外行人。由于缺乏专家资源,这些方法在资金和时间上都很昂贵。在这项工作中,我们探讨了外行是否可以预测社会科学研究的可复制性,而无需专家干预。我们向非专业人士提供了从已发表的社会科学论文中截取的原材料,并引发了他们对与论文相关的问题的回答。我们的研究结果表明,外行人从事这项技术任务,提供合理和自足的答案。他们中的大多数人也表现出对材料的良好理解。然而,征集的信息对实际复制结果的预测能力有限。我们进一步讨论了我们学到的几个教训相比,专家干预的方法,以启发未来的工作。

英文摘要:

The low replication rate of published studies has long concerned the social science community, making understanding the replicability a critical problem. Several studies have shown that relevant research communities can make predictions about the replicability of individual studies with above-chance accuracy. Follow-up work further indicates that laypeople can also achieve above-chance accuracy in predicting replicability when experts interpret the studies into short descriptions that are more accessible for laypeople. The involvement of scarce expert resources may make these methods expensive from financial and time perspectives. In this work, we explored whether laypeople can predict the replicability of social science studies without expert intervention. We presented laypeople with raw materials truncated from published social science papers and elicited their answers to questions related to the paper. Our results suggested that laypeople were engaged in this technical task, providing reasonable and self-contained answers. The majority of them also demonstrated a good understanding of the material. However, the solicited information had limited predictive power on the actual replication outcomes. We further discuss several lessons we learned compared to the approach with expert intervention to inspire future works.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑