LAB-Bench：衡量语言模型生物学研究能力的基准
LAB-Bench: Measuring Capabilities of Language Models for Biology Research
AI总结：
本文提出LAB-Bench，一个包含2400多道多项选择题的基准，用于评估语言模型在文献检索、图表解读、数据库导航及序列操作等实际生物学研究任务上的能力，并初步比较了前沿模型与人类专家的表现。
摘要：
人们普遍乐观地认为，前沿大型语言模型（LLM）和LLM增强系统有潜力迅速加速各学科的科学发现。如今，存在许多基准来衡量LLM在教科书式科学问题上的知识和推理能力，但很少有基准旨在评估语言模型在科学研究所需实际任务上的表现，如文献检索、实验方案规划和数据分析。作为构建此类基准的一步，我们引入了语言智能体生物学基准（LAB-Bench），这是一个包含超过2400道多项选择题的广泛数据集，用于评估AI系统在一系列实际生物学研究能力上的表现，包括文献的回忆与推理、图表解读、数据库的访问与导航，以及DNA和蛋白质序列的理解与操作。重要的是，与之前的科学基准相比，我们期望一个能在较难的LAB-Bench任务上持续获得高分的AI系统，能在文献检索和分子克隆等领域为研究人员提供有用的辅助。作为对前沿语言模型新兴科学任务能力的初步评估，我们测量了多个模型在我们基准上的表现，并与人类专家生物学研究者的结果进行了比较。我们将持续更新和扩展LAB-Bench，并期望它能在未来自动化研究系统的开发中成为一个有用的工具。LAB-Bench的一个公开子集可在以下网址使用：https://huggingface.co/datasets/futurehouse/lab-bench
英文摘要：
There is widespread optimism that frontier Large Language Models (LLMs) and LLM-augmented systems have the potential to rapidly accelerate scientific discovery across disciplines. Today, many benchmarks exist to measure LLM knowledge and reasoning on textbook-style science questions, but few if any benchmarks are designed to evaluate language model performance on practical tasks required for scientific research, such as literature search, protocol planning, and data analysis. As a step toward building such benchmarks, we introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of over 2,400 multiple choice questions for evaluating AI systems on a range of practical biology research capabilities, including recall and reasoning over literature, interpretation of figures, access and navigation of databases, and comprehension and manipulation of DNA and protein sequences. Importantly, in contrast to previous scientific benchmarks, we expect that an AI system that can achieve consistently high scores on the more difficult LAB-Bench tasks would serve as a useful assistant for researchers in areas such as literature search and molecular cloning. As an initial assessment of the emergent scientific task capabilities of frontier language models, we measure performance of several against our benchmark and report results compared to human expert biology researchers. We will continue to update and expand LAB-Bench over time, and expect it to serve as a useful tool in the development of automated research systems going forward. A public subset of LAB-Bench is available for use at the following URL: https://huggingface.co/datasets/futurehouse/lab-bench
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
