arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.09063q-bio.GNcs.AI
收藏

scBench:在单细胞RNA测序分析中评估AI代理

scBench: Evaluating AI Agents on Single-Cell RNA-seq Analysis

LatchBio

Kenny Workman, Zhen Yang, Harihara Muralidharan, Aidan Abdulali, Hannah Le

AI总结:

scBench通过394个可验证问题评估AI代理在单细胞RNA测序分析中的性能,揭示模型与平台选择对准确率的影响,为开发可靠分析工具提供基准。

摘要:

随着单细胞RNA测序数据集在应用、规模和复杂性方面不断增长,数据分析仍然是许多研究小组的瓶颈。尽管前沿AI代理在软件工程和一般数据分析方面有了显著进步,但尚不清楚它们是否能够从杂乱的现实单细胞数据集中提取生物学见解。我们介绍了scBench,这是一个包含394个可验证问题的基准测试,这些问题是基于涵盖六个测序平台和七个任务类别的实际scRNA-seq工作流程衍生而来的。每个问题提供了一幅实验数据在分析步骤之前的快照,并提供了一个确定性的评分器,以评估关键生物学结果的恢复情况。对八个前沿模型的基准测试数据表明,准确率范围从29-53%,存在较强的模型-任务和模型-平台交互作用。平台选择对准确性的影响与模型选择一样大,使用较少文档技术的平台会导致40多个百分点的下降。scBench与SpatialBench互补,覆盖了两种主导的单细胞模态,既作为测量工具,也作为开发能够忠实且可重复分析真实scRNA-seq数据集的代理的诊断镜。

英文摘要:

As single-cell RNA sequencing datasets grow in adoption, scale, and complexity, data analysis remains a bottleneck for many research groups. Although frontier AI agents have improved dramatically at software engineering and general data analysis, it remains unclear whether they can extract biological insight from messy, real-world single-cell datasets. We introduce scBench, a benchmark of 394 verifiable problems derived from practical scRNA-seq workflows spanning six sequencing platforms and seven task categories. Each problem provides a snapshot of experimental data immediately prior to an analysis step and a deterministic grader that evaluates recovery of a key biological result. Benchmark data on eight frontier models shows that accuracy ranges from 29-53%, with strong model-task and model-platform interactions. Platform choice affects accuracy as much as model choice, with 40+ percentage point drops on less-documented technologies. scBench complements SpatialBench to cover the two dominant single-cell modalities, serving both as a measurement tool and a diagnostic lens for developing agents that can analyze real scRNA-seq datasets faithfully and reproducibly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑