arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.04075cs.AI
收藏

你的模型理解基因吗?面向生物模型与文本模型的基因属性基准

Does your model understand genes? A benchmark of gene properties for biological and text models

IBM Research - Israel(IBM研究院以色列分部)

Yoav Kan-Tor, Michael Morris Danziger, Eden Zohar, Matan Ninio, Yishai Shimoni

AI总结:

本文提出与架构无关的基因属性基准,通过实体表示向量训练简单预测器,统一评测文本、蛋白质、DNA、表达模型及传统基线在数百个基因任务上的表现。

摘要:

近年来,深度学习方法,尤其是基础模型,在生物学研究中的应用迅速增长。这些模型可以基于文本,也可以在底层生物数据(特别是各类组学数据)上训练。然而,由于训练数据和下游任务存在差异,如何一致地比较这些模型的性能一直是一项挑战。为解决这一问题,我们开发了一种与架构无关的基准评测方法:该方法不直接评估模型,而是利用每个模型产生的实体表示向量,并为每个基准任务训练简单的预测模型。这确保所有类型的模型都使用相同的输入和输出类型接受评估。本文聚焦于从专业整理的生物信息学数据库中收集的基因属性。这些基因属性被分为五大类:基因组属性、调控功能、定位、生物过程和蛋白质属性。总体而言,我们基于这些数据库定义了数百个任务,包括二分类、多标签分类和多类分类任务。我们应用这些基准任务评估基于表达的模型、大型语言模型、蛋白质语言模型、基于 DNA 的模型以及传统基线。研究结果表明,基于文本的模型和蛋白质语言模型在基因组属性与调控功能任务上通常优于基于表达的模型,而基于表达的模型在定位任务中表现出更优性能。这些结果应有助于为生物理解和治疗发现制定更具依据的人工智能策略。为确保研究结果的可复现性和透明度,我们已将源代码和基准数据公开,供进一步研究与扩展,地址为 github.com/BiomedSciAI/gene-benchmark。

英文摘要:

The application of deep learning methods, particularly foundation models, in biological research has surged in recent years. These models can be text-based or trained on underlying biological data, especially omics data of various types. However, comparing the performance of these models consistently has proven to be a challenge due to differences in training data and downstream tasks. To tackle this problem, we developed an architecture-agnostic benchmarking approach that, instead of evaluating the models directly, leverages entity representation vectors from each model and trains simple predictive models for each benchmarking task. This ensures that all types of models are evaluated using the same input and output types. Here we focus on gene properties collected from professionally curated bioinformatics databases. These gene properties are categorized into five major groups: genomic properties, regulatory functions, localization, biological processes, and protein properties. Overall, we define hundreds of tasks based on these databases, which include binary, multi-label, and multi-class classification tasks. We apply these benchmark tasks to evaluate expression-based models, large language models, protein language models, DNA-based models, and traditional baselines. Our findings suggest that text-based models and protein language models generally outperform expression-based models in genomic properties and regulatory functions tasks, whereas expression-based models demonstrate superior performance in localization tasks. These results should aid in the development of more informed artificial intelligence strategies for biological understanding and therapeutic discovery. To ensure the reproducibility and transparency of our findings, we have made the source code and benchmark data publicly accessible for further investigation and expansion at github.com/BiomedSciAI/gene-benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑