arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12747cs.HC
收藏

从指南到实践:评估计算社会科学中方法的可重现性

From Guidelines to Practice: Evaluating the Reproducibility of Methods in Computational Social Science

Gesis - Leibniz Institute for Social Sciences(GESIS-莱布尼茨社会科学研究所)

Fakhri Momeni, Sarah Sajid, Johannes Kiesel

AI总结:

本研究评估了计算社会科学中方法的可重现性,发现整理文档和标准化执行环境能显著提高可重现性,同时参与者常依赖AI工具解决技术问题。

摘要:

可重现性仍然是计算社会科学中的核心挑战,其中复杂的流程、不断演变的软件生态系统和不一致的文档阻碍了研究人员重新执行已发表的方法。本研究系统地评估了三种条件下的可重现性:未经整理的文档、整理过的文档以及整理过的文档配以预设的执行环境。利用47次可用性测试会话,我们将行为性能指标(成功率、任务时间以及错误特征)与问卷数据和主题分析相结合,以识别技术性和概念性的可重现性障碍。整理过的文档显著减少了仓库级别的错误,并提高了用户解读方法输出的能力。标准化执行环境进一步提高了可重现性,产生了最高的成功率和最短的任务完成时间。在所有条件下,参与者经常依赖AI工具进行故障排除,通常能够独立解决这些问题而无需引导者干预。我们的发现表明,可重现性障碍是多层次的,需要协调改进文档质量、环境稳定性和概念清晰度。我们讨论了对计算社会科学中可重现性平台和基础设施设计的影响。

英文摘要:

Reproducibility remains a central challenge in computational social science, where complex workflows, evolving software ecosystems, and inconsistent documentation hinder researchers ability to re-execute published methods. This study presents a systematic evaluation of reproducibility across three conditions: uncurated documentation, curated documentation, and curated documentation paired with a preset execution environment. Using 47 usability test sessions, we combine behavioral performance indicators (success rates, task time, and error profiles) with questionnaire data and thematic analysis to identify technical and conceptual barriers to reproducibility. Curated documentation substantially reduced repository-level errors and improved users ability to interpret method outputs. Standardizing the execution environment further improved reproducibility, yielding the highest success rate and shortest task completion times. Across conditions, participants frequently relied on AI tools for troubleshooting, often enabling independent resolution of issues without facilitator intervention. Our findings demonstrate that reproducibility barriers are multi-layered and require coordinated improvements in documentation quality, environment stability, and conceptual clarity. We discuss implications for the design of reproducibility platforms and infrastructure in computational social science.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑