神经符号学文献中6.5%的内容可通过已发布的制品复现：一个六阶段审计框架及首次实例化
6.5% of the Neuro-Symbolic Literature Can Be Reproduced from Its Published Artifacts, a Six-Stage Audit Framework and First Instantiation
浏览论文内容
中文总结 AI 辅助
本文提出计算机科学领域研究可复现性审计的六阶段框架，将其应用于NSAI子领域审计后发现仅6.5%的研究可复现，呼吁强制要求实证NSAI论文提交时提供完整归档的制品包。
中文摘要 AI 辅助
我们提出了一个适用于计算机科学领域研究文献中科学主张可复现性审计的六阶段框架，并将该框架应用于神经符号人工智能（NSAI）子领域。在NSAI子领域实例化该框架后，开展了一项为期多年的审计工作：第一阶段检索到5497条记录，去除3018条重复记录；第二阶段对2479条唯一记录的标题和摘要进行筛选，识别出1365条自标识为NSAI的记录，随后因偏离主题、非研究类、无定量评估或全文不可获取的原因，在全文阶段进一步去除61条记录；第三阶段为1304条符合条件的记录寻找可验证的公开代码制品，其中849条无对应制品，剩余455条进入制品清单，并进行第四和第五阶段的受限复现。我们对85项研究实现了完全或部分复现，占符合条件语料库的6.52%，占尝试复现的18.68%。我们发现，321次尝试复现因缺少非代码制品受阻，42次因代码仓库缺失或不可用受阻。这些数据量化了即使在名义上声明“代码可用”的情况下依然存在的持续可复现性缺陷，表明未来NSAI出版物需要强制要求提供版本可控且永久归档的制品包。我们主张，实证类NSAI论文在提交时应被要求提供完整、版本可控且永久归档的制品包。
英文摘要
We present a six-stage framework for auditing the reproducibility of scientific claims across a research literature within the computer science domain, and instantiate our framework for the neuro-symbolic AI (NSAI) subdomain. Instantiating the framework on the NSAI subdomain produced a multi-year audit. Stage one retrieved 5,497 records and removed 3,018 duplicates. Stage two screened the 2,479 unique records at title and abstract, identifying 1,365 self-identified NSAI records, then removed a further 61 at full text for off-topic, non-research, no-quantitative-evaluation, or inaccessible-full-text reasons. Stage three sought a verifiable public code artifact for each of the 1,304 eligible records and found none for 849, leaving 455 to enter the artifact inventory and bounded rerun of stages four and five. We fully or partially reproduced 85 studies, 6.52% of the eligible corpus and 18.68% of attempted reruns. We found that 321 attempted reruns were blocked by missing non- code artifacts and 42 by missing or unusable code repositories. These figures quantify a persistent reproducibility deficit that survives even nominal "code available" declarations, and signal the need for enforced, versioned, and permanently archived artifact bundles in future NSAI publications. We argue that empirical NSAI papers should be required at submission time to provide complete, versioned, and permanently archived artifact bundles.
发表机构
- Department of Computer Science, University of Maryland（马里兰大学计算机科学系）
机构由 AI 辅助整理，请以论文原文为准。
