PerturBench:细胞扰动分析机器学习模型的基准测试
PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis
AI总结:
本文提出PerturBench框架,用于标准化单细胞扰动响应模型的基准测试,通过多数据集和指标评估发现简单架构更具竞争力,并强调排名指标的重要性。
摘要:
我们引入了一个用于建模单细胞转录组对扰动响应的综合框架,旨在标准化这一快速发展的领域中的基准测试。我们的方法包括一个模块化且用户友好的模型开发与评估平台、一组多样化的扰动数据集,以及一套旨在公平比较模型并剖析其性能的指标。通过对不同数据集上已发表模型和基线模型的广泛评估,我们突出了广泛使用的模型(如模式坍缩)的局限性。我们还证明了排名指标的重要性,这些指标补充了传统的模型拟合度量(如RMSE),用于验证模型有效性。值得注意的是,我们的结果表明,虽然没有任何单一模型架构明显优于其他模型,但更简单的架构通常具有竞争力,并且能很好地随更大数据集扩展。总体而言,这项基准测试工作为模型评估设立了新标准,支持了稳健的模型开发,并促进了这些模型在模拟遗传和化学筛选以用于治疗发现方面的应用。
英文摘要:
We introduce a comprehensive framework for modeling single cell transcriptomic responses to perturbations, aimed at standardizing benchmarking in this rapidly evolving field. Our approach includes a modular and user-friendly model development and evaluation platform, a collection of diverse perturbational datasets, and a set of metrics designed to fairly compare models and dissect their performance. Through extensive evaluation of both published and baseline models across diverse datasets, we highlight the limitations of widely used models, such as mode collapse. We also demonstrate the importance of rank metrics which complement traditional model fit measures, such as RMSE, for validating model effectiveness. Notably, our results show that while no single model architecture clearly outperforms others, simpler architectures are generally competitive and scale well with larger datasets. Overall, this benchmarking exercise sets new standards for model evaluation, supports robust model development, and furthers the use of these models to simulate genetic and chemical screens for therapeutic discovery.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。