LABBench2：一种改进的AI系统进行生物研究的基准测试
LABBench2: An Improved Benchmark for AI Systems Performing Biology Research
AI总结：
本文提出LABBench2，一个包含近1900个任务的改进基准，用于评估AI在真实世界中执行科学任务的能力，展示了当前前沿模型在不同子任务上的性能差异，并提供了数据集和评估工具以促进社区使用。
摘要：
对利用AI加速科学发现的乐观态度持续增长。当前AI在科学研究中的应用范围从在科学数据上训练专用基础模型到代理自主假设生成系统，再到AI驱动的自主实验室。测量AI系统在科学领域进展的需要不仅必须加快，而且越来越多地转向更现实世界的能力。除了机械知识和仅仅推理，甚至实际衡量执行有意义工作的能力。先前工作引入了语言代理生物基准LAB-Bench作为衡量这些能力的初步尝试。本文介绍了一个该基准的进化版LABBench2，用于衡量AI系统在执行有用科学任务中的现实世界能力。LABBench2包含近1900个任务，基本上是LAB-Bench的延续，测量相似的能力，但更现实的上下文。我们评估了当前前沿模型的性能，并显示虽然LAB-Bench和LABBench2测量的能力显著提高，但LABBench2在难度上提供了有意义的跳跃（模型特定的准确率差异范围从-26%到-46%在子任务上），并强调了继续改进的潜力。LABBench2继续作为AI科学研究能力的事实基准，并希望它继续帮助推进这些核心研究功能的AI工具开发。为了促进社区使用和开发，我们提供了任务数据集在https://huggingface.co/datasets/futurehouse/labbench2和公共评估工具在https://github.com/EdisonScientific/labbench2。
英文摘要：
Optimism for accelerating scientific discovery with AI continues to grow. Current applications of AI in scientific research range from training dedicated foundation models on scientific data to agentic autonomous hypothesis generation systems to AI-driven autonomous labs. The need to measure progress of AI systems in scientific domains correspondingly must not only accelerate, but increasingly shift focus to more real-world capabilities. Beyond rote knowledge and even just reasoning to actually measuring the ability to perform meaningful work. Prior work introduced the Language Agent Biology Benchmark LAB-Bench as an initial attempt at measuring these abilities. Here we introduce an evolution of that benchmark, LABBench2, for measuring real-world capabilities of AI systems performing useful scientific tasks. LABBench2 comprises nearly 1,900 tasks and is, for the most part, a continuation of LAB-Bench, measuring similar capabilities but in more realistic contexts. We evaluate performance of current frontier models, and show that while abilities measured by LAB-Bench and LABBench2 have improved substantially, LABBench2 provides a meaningful jump in difficulty (model-specific accuracy differences range from -26% to -46% across subtasks) and underscores continued room for performance improvement. LABBench2 continues the legacy of LAB-Bench as a de facto benchmark for AI scientific research capabilities and we hope that it continues to help advance development of AI tools for these core research functions. To facilitate community use and development, we provide the task dataset at https://huggingface.co/datasets/futurehouse/labbench2 and a public eval harness at https://github.com/EdisonScientific/labbench2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
