SciCoQA:科学论文与代码对齐的质量保障
SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
AI总结:
本文提出SciCoQA数据集,用于评估LLM在科学论文与代码对齐任务中的表现,揭示自动化科学质量保障的关键差距。
摘要:
科学论文与其代码之间的不一致会破坏可重复性,随着自动化研究代理生成的科学输出超出人类审查能力,这一问题愈发严峻。LLM能否可靠检测此类不一致尚未系统测量。为此,我们提出了SciCoQA数据集,包含635个论文-代码不一致案例(92个真实案例,543个合成案例),用于跨模态验证任务。在22个评估模型中,即使表现最佳的LLM(Gemini 3.1 Pro和GPT-5 Mini)也只能检测到46.7%的真实不一致案例,揭示了自动化科学质量保障的关键缺口。我们从GitHub问题和可重复性论文构建SciCoQA,并提出合成生成流程以扩展至物理、定量生物学等计算科学领域。我们进一步引入不一致类型和类别的分类体系,以表征出现的不匹配。我们的分析表明,模型在遗漏论文细节、长上下文输入以及超出其预训练语料库的论文方面尤其困难。
英文摘要:
Discrepancies between scientific papers and their code undermine reproducibility, a concern that grows as automated research agents scale scientific output beyond human review capacity. Whether LLMs can reliably detect such discrepancies has not been systematically measured. To this end, we present SciCoQA, a dataset of 635 paper-code discrepancies (92 real, 543 synthetic) for this cross-modal verification task. Across 22 evaluated models, even the best-performing LLMs, Gemini 3.1 Pro and GPT-5 Mini, detect only 46.7% of real-world discrepancies, revealing a critical gap in automated scientific quality assurance. We construct SciCoQA from GitHub issues and reproducibility papers, and propose a synthetic generation pipeline to scale beyond AI to Physics, Quantitative Biology, and other computational sciences. We further introduce a taxonomy of discrepancy types and categories to characterize the occurring mismatches. Our analysis shows that models particularly struggle with omitted paper details, long-context inputs, and papers outside their pre-training corpus.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。