知识注入的自注意力变换器
Knowledge-Infused Self Attention Transformers
AI总结：
本文提出一种模块化框架，系统地将外部知识注入变换器的自注意力、编码器层等组件，并在GLUE基准上实验验证，以缓解幻觉和对齐问题。
摘要：
基于变换器的语言模型因其使用自注意力机制捕获复杂依赖关系和上下文信息的能力，在各种自然语言处理任务中取得了显著成功。然而，它们并非没有局限性。这些局限性包括幻觉，即模型以高置信度产生错误输出；以及对齐问题，即模型为人类用户生成无益且不安全的输出。这些局限性源于数据本身缺乏隐式和缺失的上下文。为了解决这一问题，研究者们探索了从知识图谱中引入外部知识来增强这些模型，以提供所需的额外上下文。然而，现有方法的临时性使得难以恰当地分析知识注入对变换器众多可变动部分或组件的影响。本文提出了一种系统性的方法，将知识注入到基于变换器的模型的不同组件中。提出了一个模块化框架，用于识别变换器架构中的特定组件，如自注意力机制、编码器层或输入嵌入层，在这些组件上可以应用知识注入。此外，在通用语言理解评估（GLUE）基准任务上进行了大量实验，并报告了实验结果。这种系统性的方法旨在促进将知识纳入语言模型架构的更有原则性的方法。
英文摘要：
Transformer-based language models have achieved impressive success in various natural language processing tasks due to their ability to capture complex dependencies and contextual information using self-attention mechanisms. However, they are not without limitations. These limitations include hallucinations, where they produce incorrect outputs with high confidence, and alignment issues, where they generate unhelpful and unsafe outputs for human users. These limitations stem from the absence of implicit and missing context in the data alone. To address this, researchers have explored augmenting these models with external knowledge from knowledge graphs to provide the necessary additional context. However, the ad-hoc nature of existing methods makes it difficult to properly analyze the effects of knowledge infusion on the many moving parts or components of a transformer. This paper introduces a systematic method for infusing knowledge into different components of a transformer-based model. A modular framework is proposed to identify specific components within the transformer architecture, such as the self-attention mechanism, encoder layers, or the input embedding layer, where knowledge infusion can be applied. Additionally, extensive experiments are conducted on the General Language Understanding Evaluation (GLUE) benchmark tasks, and the findings are reported. This systematic approach aims to facilitate more principled approaches to incorporating knowledge into language model architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
