MetaBench:用于评估代谢组学中大型语言模型的多任务基准
MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics
AI总结:
MetaBench是首个代谢组学多任务基准,评估25个LLM在知识、理解、接地、推理和研究五种能力上的表现,发现跨库标识符接地和长尾代谢物处理仍是挑战。
摘要:
大型语言模型(LLMs)在通用文本上展现了卓越的能力;然而,它们在需要深度、相互关联知识的专业科学领域中的熟练程度在很大程度上仍未得到表征。代谢组学因其复杂的生化途径、异构的标识符系统和碎片化的数据库而呈现出独特的挑战。为了系统地评估该领域中LLM的能力,我们引入了MetaBench,这是首个用于代谢组学评估的基准。MetaBench从权威的公共资源中精选数据,评估代谢组学研究所需的五种能力:知识、理解、接地、推理和研究。我们对25个开源和闭源LLM的评估揭示了它们在代谢组学任务中的不同表现模式:虽然模型在文本生成任务上表现良好,但跨数据库的标识符接地即使在检索增强的情况下仍然具有挑战性。模型性能在具有稀疏注释的长尾代谢物上也有所下降。通过MetaBench,我们为开发和评估代谢组学AI系统提供了必要的基础设施,使代谢组学研究可靠计算工具的系统性进展成为可能。
英文摘要:
Large Language Models (LLMs) have demonstrated remarkable capabilities on general text; however, their proficiency in specialized scientific domains that require deep, interconnected knowledge remains largely uncharacterized. Metabolomics presents unique challenges with its complex biochemical pathways, heterogeneous identifier systems, and fragmented databases. To systematically evaluate LLM capabilities in this domain, we introduce MetaBench, the first benchmark for metabolomics assessment. Curated from authoritative public resources, MetaBench evaluates five capabilities essential for metabolomics research: knowledge, understanding, grounding, reasoning, and research. Our evaluation of 25 open- and closed-source LLMs reveals distinct performance patterns across metabolomics tasks: while models perform well on text generation tasks, cross-database identifier grounding remains challenging even with retrieval augmentation. Model performance also decreases on long-tail metabolites with sparse annotations. With MetaBench, we provide essential infrastructure for developing and evaluating metabolomics AI systems, enabling systematic progress toward reliable computational tools for metabolomics research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。