arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.04515cs.CL
收藏

一种带有栈注意力的Transformer

A Transformer with Stack Attention

ETH Zürich(苏黎世联邦理工学院);University of Cambridge(剑桥大学)

Jiaoda Li, Jennifer C. White, Mrinmaya Sachan, Ryan Cotterell

AI总结:

针对Transformer无法建模许多上下文无关语言任务的局限,提出为其增加可微的基于栈的注意力机制,该机制可集成到任意Transformer语言模型中并提升可解释性,能让Transformer建模部分确定性上下文无关语言。

摘要:

自然语言被认为是(轻度)上下文相关的。尽管Transformer为能力出众的大语言模型提供了基础支撑,但它无法对许多上下文无关语言任务建模。为解决基于Transformer的语言模型在建模能力上的这一局限,我们提出为其增加一种可微的、基于栈的注意力机制。该机制可被集成到任何基于Transformer的语言模型中,并能为模型增添一定程度的可解释性。我们的实验表明,加入这种基于栈的注意力机制后,Transformer能够对部分(而非全部)确定性上下文无关语言进行建模。

英文摘要:

Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition of our stack-based attention mechanism enables the transformer to model some, but not all, deterministic context-free languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑