ATLANTIC：面向跨学科科学的结构感知检索增强语言模型
ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Science
AI总结：
提出结构感知检索增强语言模型ATLANTIC，通过构建跨学科异构文档图并利用图神经网络融合段落文本与结构嵌入，提升了科学任务中检索文档的连贯性、忠实度与相关性。
摘要：
大型语言模型在许多自然语言处理任务上取得了令人瞩目的性能。然而，它们的知识容量受限于预训练语料库。检索增强通过从外部知识源检索上下文以补充语言模型，提供了一种有效的解决方案。然而，现有的检索增强技术忽略了这些文档之间的结构关系。此外，检索模型在科学任务中鲜少被探索，尤其是在检索文档的忠实度方面。在本文中，我们提出了一种新颖的结构感知检索增强语言模型，该模型在检索增强过程中纳入了文档结构。我们创建了一个异构文档图，捕获连接超过15个科学学科（例如物理学、医学、化学等）文档的多种关系类型（例如引用、合著等）。我们在精选的文档图上训练图神经网络，作为模型预训练期间检索到的对应段落的结构编码器。特别地，连同检索段落的文本嵌入，我们获取了文档（段落）的结构嵌入，并在将其输入语言模型之前将它们融合在一起。我们在包括科学问答和科学文档分类任务在内的各种科学基准上广泛评估了我们的模型。实验结果表明，结构感知检索改善了检索到的更连贯、忠实且上下文相关的段落，同时在整体准确率上表现出可比的性能。
英文摘要：
Large language models record impressive performance on many natural language processing tasks. However, their knowledge capacity is limited to the pretraining corpus. Retrieval augmentation offers an effective solution by retrieving context from external knowledge sources to complement the language model. However, existing retrieval augmentation techniques ignore the structural relationships between these documents. Furthermore, retrieval models are not explored much in scientific tasks, especially in regard to the faithfulness of retrieved documents. In this paper, we propose a novel structure-aware retrieval augmented language model that accommodates document structure during retrieval augmentation. We create a heterogeneous document graph capturing multiple types of relationships (e.g., citation, co-authorship, etc.) that connect documents from more than 15 scientific disciplines (e.g., Physics, Medicine, Chemistry, etc.). We train a graph neural network on the curated document graph to act as a structural encoder for the corresponding passages retrieved during the model pretraining. Particularly, along with text embeddings of the retrieved passages, we obtain structural embeddings of the documents (passages) and fuse them together before feeding them to the language model. We evaluate our model extensively on various scientific benchmarks that include science question-answering and scientific document classification tasks. Experimental results demonstrate that structure-aware retrieval improves retrieving more coherent, faithful and contextually relevant passages, while showing a comparable performance in the overall accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
