arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.05565cs.LGcs.CL
收藏

用于高效实体追踪的链式注意力与因果注意力

Chain and Causal Attention for Efficient Entity Tracking

LAMSADE(拉姆萨德实验室);Université Paris Dauphine-PSL(巴黎多芬大学-PSL)

Erwan Fagnou, Paul Caillon, Blaise Delattre, Alexandre Allauzen

AI总结:

该研究指出Transformer处理实体追踪存在层数约束,提出将注意力视为邻接矩阵的增强机制,仅单层即可追踪实体状态,在相关数据集上效果显著提升且减少了所需层数。

摘要:

本文研究了大语言模型中Transformer在实体追踪任务上的局限性。我们发现了一项理论约束:对于包含n次状态变化的实体追踪任务,Transformer至少需要log₂(n+1)层才能处理。为解决该问题,我们对标准注意力机制提出了一种高效精简的增强方案,使其能更高效地处理长距离依赖。通过将注意力视为邻接矩阵,我们的模型仅需单层即可追踪实体状态。实验结果表明,该方法在实体追踪数据集上取得了显著提升,同时在标准自然语言建模任务上保持了具有竞争力的性能。我们改进后的注意力机制能用大幅减少的层数达到同等性能,此外还展现出结构化的注意力内部表征。在玩具数据集与复杂数据集上的大量实验验证了我们的方法。本文的贡献包括理论洞见、改进的注意力机制以及实证验证。

英文摘要:

This paper investigates the limitations of transformers for entity-tracking tasks in large language models. We identify a theoretical constraint, showing that transformers require at least $\log_2 (n+1)$ layers to handle entity tracking with $n$ state changes. To address this issue, we propose an efficient and frugal enhancement to the standard attention mechanism, enabling it to manage long-term dependencies more efficiently. By considering attention as an adjacency matrix, our model can track entity states with a single layer. Empirical results demonstrate significant improvements in entity tracking datasets while keeping competitive performance on standard natural language modeling. Our modified attention allows us to achieve the same performance with drastically fewer layers. Additionally, our enhanced mechanism reveals structured internal representations of attention. Extensive experiments on both toy and complex datasets validate our approach. Our contributions include theoretical insights, an improved attention mechanism, and empirical validation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑