语法引导的稀疏注意力：高效且可解释的Transformer
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
AI总结：
提出语法引导的稀疏注意力方法，通过词性标签动态生成注意力掩码，在保持精度的同时降低计算复杂度。
摘要：
Transformer模型中自注意力的二次复杂度仍然是处理长序列和高效部署大型语言模型的主要瓶颈。为此，已有大量关于稀疏注意力的研究，Deepseek稀疏注意力结合了多种创建令牌片段的方法以降低时间复杂度。本文提出了一种新颖的方法——语法引导的稀疏注意力，它基于令牌的语法角色约束注意力计算。通过利用词性（POS）标签，动态生成注意力掩码，强制令牌之间建立语言上连贯的连接，从而在不牺牲必要语言依赖性的情况下减少计算图。提出并评估了两种掩码策略：硬掩码严格只允许预定义的语法交互，软掩码则将注意力偏向这些交互。使用类似DistilBERT的架构在SST-2情感分类任务上进行的实验表明，语法引导的稀疏注意力在保持与全注意力相当的精度的同时，显著降低了理论计算开销。初步结果显示，硬掩码的准确率为0.8200，软掩码为0.8165，与全注意力的0.8200非常接近，为构建更高效、可解释且具有语言知识的Transformer架构提供了途径。
英文摘要：
The quadratic complexity of self-attention in Transformer models remains a significant bottleneck for processing long sequences and deploying large language models efficiently. For this approach, there has been significant research into Sparse Attention, and Deepseek Sparse Attention has combined various methods of creating segments of tokens to reduce the time complexity. This paper introduces a novel approach, Grammatically-Guided Sparse Attention, which constrains attention computations based on the grammatical roles of tokens. By leveraging Parts-of-Speech (POS) tags, attention masks are dynamically generated that enforce linguistically coherent connections between tokens, reducing the computational graph without sacrificing essential linguistic dependencies. Two masking strategies are proposed and evaluated: a hard mask that strictly allows only predefined grammatical interactions, and a soft mask that biases attention towards these interactions. The experiments, conducted on the SST-2 sentiment classification task using a DistilBERT-like architecture, demonstrate that Grammatically-Guided Sparse Attention maintains comparable accuracy to full attention while significantly reducing the theoretical computational overhead. Preliminary results show accuracy values of 0.8200 for hard masking and 0.8165 for soft masking, closely matching the 0.8200 of full attention, providing a path towards more efficient, interpretable, and linguistically-informed Transformer architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
