arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.09667cs.CLcs.AIq-bio.GN
收藏

GeneGPT:利用领域工具增强大语言模型以改善生物医学信息获取

GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information

National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆);University of Maryland, College Park(马里兰大学帕克分校)

Qiao Jin, Yifan Yang, Qingyu Chen, Zhiyong Lu

AI总结:

GeneGPT 通过上下文学习和增强解码算法教大语言模型调用 NCBI Web API 回答基因组学问题,在 GeneTuring 基准上以平均 0.83 分显著超越检索增强和生物医学大语言模型。

摘要:

尽管大语言模型(LLMs)已成功应用于各种任务,但它们仍面临幻觉问题。用数据库工具等特定领域工具增强大语言模型,可以促进更便捷、更精确地获取专业知识。在本文中,我们提出了 GeneGPT,一种教大语言模型使用美国国家生物技术信息中心(NCBI)的 Web API 来回答基因组学问题的新方法。具体来说,我们通过上下文学习和一种能够检测并执行 API 调用的增强解码算法,提示 Codex 使用 NCBI Web API 解决 GeneTuring 测试。实验结果表明,GeneGPT 在 GeneTuring 基准的八个任务上取得了最先进的性能,平均得分为 0.83,大幅超越了检索增强的大语言模型,如新版 Bing(0.44)、生物医学大语言模型如 BioMedLM(0.08)和 BioGPT(0.04),以及 GPT-3(0.16)和 ChatGPT(0.12)。我们的进一步分析表明:(1)API 演示具有良好的跨任务泛化能力,并且对于上下文学习比文档更有用;(2)GeneGPT 可以泛化到更长的 API 调用链,并回答 GeneHop 中的多跳问题,GeneHop 是本文引入的一个新数据集;(3)不同类型的错误在不同任务中富集,为未来的改进提供了有价值的见解。

英文摘要:

While large language models (LLMs) have been successfully applied to various tasks, they still face challenges with hallucinations. Augmenting LLMs with domain-specific tools such as database utilities can facilitate easier and more precise access to specialized knowledge. In this paper, we present GeneGPT, a novel method for teaching LLMs to use the Web APIs of the National Center for Biotechnology Information (NCBI) for answering genomics questions. Specifically, we prompt Codex to solve the GeneTuring tests with NCBI Web APIs by in-context learning and an augmented decoding algorithm that can detect and execute API calls. Experimental results show that GeneGPT achieves state-of-the-art performance on eight tasks in the GeneTuring benchmark with an average score of 0.83, largely surpassing retrieval-augmented LLMs such as the new Bing (0.44), biomedical LLMs such as BioMedLM (0.08) and BioGPT (0.04), as well as GPT-3 (0.16) and ChatGPT (0.12). Our further analyses suggest that: (1) API demonstrations have good cross-task generalizability and are more useful than documentations for in-context learning; (2) GeneGPT can generalize to longer chains of API calls and answer multi-hop questions in GeneHop, a novel dataset introduced in this work; (3) Different types of errors are enriched in different tasks, providing valuable insights for future improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑