BioProBench: 一个全面的数据集和生物协议理解与推理基准
BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science
AI总结：
BioProBench通过构建大规模数据集和基准，提升生物协议理解与推理的LLM性能。
摘要：
自主科学实验的实现目前受到LLM在理解生物协议所需的严格程序逻辑和准确性方面的困难限制。为解决这一根本性挑战，我们提出了BioProBench，一个用于生物学程序推理的综合性资源。BioProBench基于BioProCorpus，一个包含27,000个人工编写协议的基础集合。从该语料库中，我们系统地构建了一个包含超过550,000个任务实例的数据集，提供了大规模训练资源和具有新指标的严格基准。评估10种主流LLM后发现，虽然通用理解能力较高，但在需要深度推理、定量精度和安全意识的任务上表现显著下降。为了展示BioProCorpus在缓解这些问题的价值，我们开发了ProAgent，其基于我们的语料库，ProAgent显著提升了现有技术水平。BioProBench提供了一个严谨的诊断基准和开发下一代可靠科学AI的基础资源。代码和数据可在：https://github.com/YuyangSunshine/bioprotocolbench 和 https://huggingface.co/datasets/BioProBench/BioProBench 获得。
英文摘要：
The realization of autonomous scientific experimentation is currently limited by LLMs' struggle to grasp the strict procedural logic and accuracy required by biological protocols. To address this fundamental challenge, we present \textbf{BioProBench}, a comprehensive resource for procedural reasoning in biology. BioProBench is grounded in \textbf{BioProCorpus}, a foundational collection of 22,413 human-written protocols. From this corpus, we systematically constructed a dataset of 523,784 task instances, offering both a large-scale training resource and a rigorous benchmark with novel metrics. Evaluating 10 mainstream LLMs, we find that while general comprehension is high, performance drops significantly on tasks demanding deep reasoning, quantitative precision, and safety awareness. To demonstrate the value of BioProCorpus in mitigating these issues, we developed \textbf{ProAgent}, grounded in our corpus, ProAgent substantially advances the state-of-the-art. https://github.com/YuyangSunshine/bioprobench and https://huggingface.co/BioProBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
