CellChain:面向扰动推理的跨尺度证据使用基准
CellChain: Benchmarking Cross-Scale Evidence Use for Perturbation Reasoning
AI总结:
本研究推出CellChain基准,涵盖三类生物任务的大量案例与证据实例,评估18个基础模型的跨尺度扰动推理能力,发现证据覆盖与准确率无必然关联,且匹配参考历史可提升药物敏感性预测性能。
摘要:
近期,基础模型与大规模细胞扰动数据集的进展,推动了利用大型语言模型(LLMs)预测和推理细胞响应的兴趣增长。该场景下的生物推理涉及跨尺度关联分子机制、细胞响应与功能结果,因此评估基础模型不仅需判断其预测是否正确,还需评估这些预测如何依赖所提供的生物证据。我们推出CellChain,这是一个基于实验的基准,涵盖590个生物案例与2360个匹配的证据-条件实例,涉及基因依赖预测、细胞因子轴识别及药物敏感性排序三类任务。CellChain通过一个可追溯来源的证据图,将扰动实验与外部生物资源关联,该图涵盖分子、通路、转录及表型信息。特定任务的条件在保留问题、候选答案、参考目标与评分规则的同时,改变证据的覆盖范围、组成或分配。在18个基础模型中,预设目标-证据条件下的最高准确率分别为57.5%、75.5%和70.8%,且更广泛的证据覆盖并不总能提升准确率。在药物敏感性基准队列中,匹配的参考历史在聚类加权准确率上比交换后的历史平均高出13.0个百分点。决策转换分析区分了修正、新引入的错误及错误答案切换,揭示了小净准确率差异背后隐藏的大量答案变化。CellChain提供了受控比较,可将模型预测的变化归因于生物上下文的预设扩展,包括证据覆盖、组成及生物分配的变化。
英文摘要:
Recent advances in foundation models and large-scale cellular perturbation datasets have driven growing interest in using LLMs to predict and reason about cellular responses. Biological reasoning in this setting involves relating molecular mechanisms, cellular responses, and functional outcomes across scales. Evaluating foundation models therefore requires assessing not only whether their predictions are correct, but also how those predictions depend on the biological evidence provided. We introduce CellChain, an experimentally grounded benchmark comprising 590 biological cases and 2,360 matched evidence-condition instances across gene-dependency prediction, cytokine-axis identification, and drug-sensitivity ranking. CellChain links perturbation experiments and external biological resources through a provenance-aware evidence graph spanning molecular, pathway, transcriptional, and phenotypic information. Task-specific conditions vary evidence coverage, composition, or assignment while preserving questions, candidate answers, reference targets, and scoring rules. Across 18 foundation models, the highest accuracies under prespecified target-evidence conditions are 57.5%, 75.5%, and 70.8%, respectively, and broader evidence coverage does not consistently improve accuracy. On the drug-sensitivity benchmark cohort, matched reference histories outperform swapped histories by an average of 13.0 percentage points in cluster-weighted accuracy. Decision-transition analysis separates corrections, newly introduced errors, and wrong-answer switches, revealing substantial answer changes hidden by small net accuracy differences. CellChain provides controlled comparisons for attributing changes in model predictions to prespecified expansions of biological context, including changes in evidence coverage, composition, and biological assignment.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。