arXiv 2405.081512025-11-14cs.CL

生物医学自然语言处理中检索增强大语言模型的基准测试：应用、鲁棒性与自我感知

Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness

Division of Computational Health Sciences, Department of Surgery University of Minnesota（大学医学计算科学系，外科系）；Institute for Health Informatics University of Minnesota（健康信息学研究所，明尼苏达大学）；Department of Electrical and Computer Engineering University of Minnesota（电气与计算机工程系，明尼苏达大学）

Mingchen Li, Zaifu Zhan, Han Yang, Yongkang Xiao, Jiatan Huang, Rui Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文针对生物医学NLP场景，构建评估框架与测试平台，系统测评搭配不同检索器的代表性RAL在5类任务上的应用表现、三类鲁棒性及自我感知能力。

摘要：

大语言模型（LLM）凭借输入上下文中的示例适配新任务，已在各类生物医学自然语言处理（NLP）任务中展现出卓越能力，但LLM对示例选择较为敏感。为解决LLM固有的幻觉问题，检索增强大语言模型（RAL）通过从已有数据库中检索相关信息提供了解决方案。然而现有研究尚未严格评估检索增强大语言模型在不同生物医学NLP任务上的效果，这一不足使得人们难以确定RAL在生物医学领域的能力。此外，RAL的输出会受到检索到的无标注、反事实或多样性知识的影响，这类知识在生物医学领域尚未得到充分研究，却在真实世界中普遍存在。最后，探索自我感知能力对RAL系统也至关重要。因此本文系统研究了RAL在三元组抽取、链接预测、分类、问答、自然语言推理这5类不同生物医学任务上的效果，分析了RAL在无标注鲁棒性、反事实鲁棒性、多样性鲁棒性、负向感知这四项基础能力上的表现。为此本文提出了一套评估框架，用于测评RAL在不同生物医学NLP任务上的性能，并基于上述基础能力构建了四个不同的测试平台，随后在9个数据集的5项任务上，对搭配3种不同检索器的3个代表性LLM开展了评估。

英文摘要：

Large language models (LLM) have demonstrated remarkable capabilities in various biomedical natural language processing (NLP) tasks, leveraging the demonstration within the input context to adapt to new tasks. However, LLM is sensitive to the selection of demonstrations. To address the hallucination issue inherent in LLM, retrieval-augmented LLM (RAL) offers a solution by retrieving pertinent information from an established database. Nonetheless, existing research work lacks rigorous evaluation of the impact of retrieval-augmented large language models on different biomedical NLP tasks. This deficiency makes it challenging to ascertain the capabilities of RAL within the biomedical domain. Moreover, the outputs from RAL are affected by retrieving the unlabeled, counterfactual, or diverse knowledge that is not well studied in the biomedical domain. However, such knowledge is common in the real world. Finally, exploring the self-awareness ability is also crucial for the RAL system. So, in this paper, we systematically investigate the impact of RALs on 5 different biomedical tasks (triple extraction, link prediction, classification, question answering, and natural language inference). We analyze the performance of RALs in four fundamental abilities, including unlabeled robustness, counterfactual robustness, diverse robustness, and negative awareness. To this end, we proposed an evaluation framework to assess the RALs' performance on different biomedical NLP tasks and establish four different testbeds based on the aforementioned fundamental abilities. Then, we evaluate 3 representative LLMs with 3 different retrievers on 5 tasks over 9 datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。