利用相对位置标签将异构图与实体感知自注意力集成用于阅读理解模型
Integrating a Heterogeneous Graph with Entity-aware Self-attention using Relative Position Labels for Reading Comprehension Model
AI总结：
本文提出一种无需外部知识的注意力模式，将异构图推理知识与LUKE实体感知自注意力结合，并在ReCoRD和WikiHop上超越LUKE及LUKE-Graph模型。
摘要：
尽管Transformer模型在机器阅读理解任务中取得了显著进展，但由于输入序列中缺乏显式知识，它们在处理复杂推理任务时仍然存在不足。为了解决这一局限，许多近期工作提出将外部知识注入模型。然而，选择相关的外部知识、确保其可用性以及需要额外的处理步骤仍然具有挑战性。在本文中，我们引入了一种新颖的注意力模式，将源自异构图的推理知识集成到Transformer架构中，而无需依赖外部知识。所提出的注意力模式包含三个关键要素：用于词元（word token）的全局-局部注意力；用于实体元（entity token）的图注意力，该注意力对图中相连的元表现出强关注，而对不相连的元关注较弱；以及考虑每个实体元与词元之间关系的类型。如果存在关系，这会导致两者之间的注意力得到优化。该模式与特殊的相对位置标签相结合，使其能够与LUKE的实体感知自注意力机制集成。实验结果表明，我们的模型在两个不同的数据集上均优于最先进的LUKE-Graph和基线LUKE模型：强调常识推理的ReCoRD和专注于多跳推理挑战的WikiHop。
英文摘要：
Despite the significant progress made by transformer models in machine reading comprehension tasks, they still fall short in handling complex reasoning tasks due to the absence of explicit knowledge in the input sequence. To address this limitation, many recent works have proposed injecting external knowledge into the model. However, selecting relevant external knowledge, ensuring its availability, and requiring additional processing steps remain challenging. In this paper, we introduce a novel attention pattern that integrates reasoning knowledge derived from a heterogeneous graph into the transformer architecture without relying on external knowledge. The proposed attention pattern comprises three key elements: global-local attention for word tokens, graph attention for entity tokens that exhibit strong attention towards tokens connected in the graph as opposed to those unconnected, and the consideration of the type of relationship between each entity token and word token. This results in optimized attention between the two if a relationship exists. The pattern is coupled with special relative position labels, allowing it to integrate with LUKE's entity-aware self-attention mechanism. The experimental findings corroborate that our model outperforms both the cutting-edge LUKE-Graph and the baseline LUKE model across two distinct datasets: ReCoRD, emphasizing commonsense reasoning, and WikiHop, focusing on multi-hop reasoning challenges.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
