KSAT:知识注入的自注意力Transformer——整合多个特定领域上下文
KSAT: Knowledge-infused Self Attention Transformer -- Integrating Multiple Domain-Specific Contexts
AI总结:
提出知识注入的自注意力Transformer(KSAT),通过专用自注意力层引入外部知识引导偏置,实现多领域上下文整合,在可解释性和性能上均优于微调基线。
摘要:
特定领域的语言理解需要整合多个相关的上下文信息。例如,在文本“我有一把枪,感觉生活很糟糕,如果我明天不再醒来也不是最坏的事”中,我们看到了自杀和抑郁相关的行为(多个上下文)。自注意力架构中的领域特定性是通过对相关领域特定资源(数据集和外部知识——关于自杀和抑郁相关心理健康诊断的医学教科书章节)的摘录进行微调来处理的。我们提出了一种修改后的自注意力架构,即知识注入的自注意力Transformer(KSAT),它通过使用外部知识源来实现多个特定领域上下文的整合。KSAT在每个知识源的专用自注意力层中引入了知识引导的偏置来实现这一点。此外,KSAT提供了控制从数据学习和从知识学习之间权衡的机制。我们的定量和定性评估表明:(1)KSAT架构提供了新颖的人类可理解方式,以精确测量和可视化注入的领域上下文的贡献;(2)KSAT与其他知识注入的基线模型表现相当,并且显著优于使用微调进行特定领域任务的基线模型。
英文摘要:
Domain-specific language understanding requires integrating multiple pieces of relevant contextual information. For example, we see both suicide and depression-related behavior (multiple contexts) in the text ``I have a gun and feel pretty bad about my life, and it wouldn't be the worst thing if I didn't wake up tomorrow''. Domain specificity in self-attention architectures is handled by fine-tuning on excerpts from relevant domain specific resources (datasets and external knowledge - medical textbook chapters on mental health diagnosis related to suicide and depression). We propose a modified self-attention architecture Knowledge-infused Self Attention Transformer (KSAT) that achieves the integration of multiple domain-specific contexts through the use of external knowledge sources. KSAT introduces knowledge-guided biases in dedicated self-attention layers for each knowledge source to accomplish this. In addition, KSAT provides mechanics for controlling the trade-off between learning from data and learning from knowledge. Our quantitative and qualitative evaluations show that (1) the KSAT architecture provides novel human-understandable ways to precisely measure and visualize the contributions of the infused domain contexts, and (2) KSAT performs competitively with other knowledge-infused baselines and significantly outperforms baselines that use fine-tuning for domain-specific tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。