将结构化生物医学知识注入语言模型：持续预训练与GraphRAG
Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG
AI总结：
本文提出通过持续预训练和GraphRAG两种方法将结构化生物医学知识注入语言模型，提升其在生物医学领域的表现，实验显示BERTUMLS在知识密集型问答任务中表现优异，GraphRAG在PubMedQA和BioASQ上提升显著。
摘要：
将领域特定知识注入语言模型（LMs）对于适应生物医学等专业领域至关重要。尽管大多数现有方法依赖于非结构化文本语料库，本研究探索了两种互补策略，利用UMLS Metathesaurus中的结构化知识：（i）持续预训练，将知识嵌入模型参数中，以及（ii）Graph Retrieval-Augmented Generation（GraphRAG），在推理时咨询知识图谱。我们首先构建了一个大规模生物医学知识图谱，从UMLS中提取（340万个概念和3420万条关系），存储在Neo4j中以实现高效查询。然后从该图谱中推导出一个约10亿词的文本语料库，用于持续预训练两个模型：BERTUMLS（基于BERT）和BioBERTUMLS（基于BioBERT）。我们评估了这些模型在六个BLURB（生物医学语言理解和推理基准）数据集上的表现，涵盖五种任务类型，并评估GraphRAG在两个问答（QA）数据集（PubMedQA、BioASQ）上的表现。在BLURB任务中，BERTUMLS在BERT上有所提升，最大收益出现在知识密集型问答任务中。对BioBERT的影响更为复杂，表明当基础模型已编码大量生物医学文本知识时，收益会下降。最后，将LLaMA 3-8B与我们的GraphRAG流水线结合，可在PubMedQA上提升超过3个点，在BioASQ上提升5个点，无需重新训练，提供透明、多跳和易于更新的知识访问。我们发布了处理后的UMLS Neo4j图谱以支持可重复性。
英文摘要：
The injection of domain-specific knowledge is crucial for adapting language models (LMs) to specialized fields such as biomedicine. While most current approaches rely on unstructured text corpora, this study explores two complementary strategies for leveraging structured knowledge from the UMLS Metathesaurus: (i) Continual pretraining that embeds knowledge into model parameters, and (ii) Graph Retrieval-Augmented Generation (GraphRAG) that consults a knowledge graph at inference time. We first construct a large-scale biomedical knowledge graph from UMLS (3.4 million concepts and 34.2 million relations), stored in Neo4j for efficient querying. We then derive a ~100-million-token textual corpus from this graph to continually pretrain two models: BERTUMLS (from BERT) and BioBERTUMLS (from BioBERT). We evaluate these models on six BLURB (Biomedical Language Understanding and Reasoning Benchmark) datasets spanning five task types and evaluate GraphRAG on the two QA (Question Answering) datasets (PubMedQA, BioASQ). On BLURB tasks, BERTUMLS improves over BERT, with the largest gains on knowledge-intensive QA. Effects on BioBERT are more nuanced, suggesting diminishing returns when the base model already encodes substantial biomedical text knowledge. Finally, augmenting LLaMA 3-8B with our GraphRAG pipeline yields over than 3 points accuracy on PubMedQA and 5 points on BioASQ without any retraining, delivering transparent, multi-hop, and easily updated knowledge access. We release the processed UMLS Neo4j graph to support reproducibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
