scBench-Long:长周期单细胞生物学的可验证基准测试
scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology
AI总结:
提出scBench-Long基准,评估AI代理从原始单细胞数据中恢复科学结论的能力,涵盖21项任务,最强模型通过率仅25.4%。
摘要:
单细胞研究要求分析人员通过多步骤工作流程以及元数据、检测背景和辅助证据的整合,将原始测量结果转化为特定的生物学结论。现有的AI-生物学基准主要衡量广泛知识、可执行工作流程或局部分析步骤。我们引入了scBench-Long,一个用于长周期单细胞生物学的基准,其中代理必须从原始或接近原始的数据中恢复科学结论,而无需预设方法。该基准包含21项评估,涵盖黑色素瘤CD8 T细胞反应性、CD8 RNA+ATAC调控推断、人-猴嵌合体发育、KRAS驱动的肺肿瘤衰老以及致命性COVID-19肺部病理。任务涵盖配对scRNA/TCR测序、RNA和染色质分析、跨物种转录组学、组合scRNA-seq、单核RNA-seq、免疫组库、直系同源图谱、配体-受体资源和验证证据。候选结论经过复制、审查并转化为受控答案词汇表,具有确定性评分和轨迹评分标准。在1,068条完成的轨迹中,最强的模型-工具组合通过了16/63次运行(25.4%)。scBench-Long评估代理是否能够超越局部分析步骤,做出由单细胞数据支持的复杂科学结论。
英文摘要:
Single-cell studies require analysts to convert raw measurements into specific biological claims through multi-step workflows and integration of metadata, assay context, and auxiliary evidence. Existing AI-biology benchmarks largely measure broad knowledge, executable workflows, or local analysis steps. We introduce scBench-Long, a benchmark for long-horizon single-cell biology in which agents must recover scientific conclusions from raw or near-raw data without prescribed methods. The benchmark contains 21 evaluations spanning melanoma CD8 T-cell reactivity, CD8 RNA+ATAC regulatory inference, human--monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology. Tasks cover paired scRNA/TCR sequencing, RNA and chromatin profiling, cross-species transcriptomics, combinatorial scRNA-seq, single-nucleus RNA-seq, immune repertoires, ortholog maps, ligand--receptor resources, and validation evidence. Candidate claims are reproduced, reviewed, and converted into controlled answer vocabularies with deterministic grading and trajectory rubrics. Across 1,068 completed trajectories, the strongest model--harness pair passes 16/63 runs (25.4\%). scBench-Long evaluates whether agents can move beyond local analysis steps and make complex scientific claims that are supported by single-cell data.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。