InfiniteScienceGym：一个无界的、程序生成的科学分析基准
InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
AI总结：
本文提出InfiniteScienceGym，通过程序生成科学仓库和可验证问答任务，评估证据推理、回避和工具分析能力，发现现有模型在回答不可答问题上存在显著缺陷。
摘要：
大型语言模型正成为科学助手，但评估其从实证数据推理能力仍具挑战性。现有基准继承了出版偏见、已知知识偏见、标签噪声和存储需求。本文提出InfiniteScienceGym，通过种子生成自包含的科学仓库及可验证问答任务，实现可控评估。该基准通过程序生成科学仓库和可验证问答任务，评估证据推理、回避和工具分析能力，发现现有模型在回答不可答问题上存在显著缺陷。
英文摘要：
Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 50% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
