TensorLens: 通过高阶注意力张量实现端到端的Transformer分析
TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors
AI总结：
TensorLens通过高阶注意力张量实现端到端Transformer分析，提供统一的线性表示方法，增强模型可解释性和理解能力。
摘要：
注意力矩阵是变压器研究中的基本要素，支持包括可解释性、可视化、操控和蒸馏在内的广泛应用。然而，大多数现有分析集中在单个注意力头或层上，未能考虑模型的整体行为。尽管先前的努力已通过平均和矩阵乘法扩展了跨多个头的注意力公式，或纳入了诸如归一化和FFN等组件，但仍然缺乏一个统一且完整的表示方法，能够涵盖所有Transformer块。我们通过引入TensorLens，一种新的公式，将整个Transformer表示为一个输入依赖的线性算子，通过高阶注意力交互张量表达。该张量共同编码了注意力、FFN、激活、归一化和残差连接，提供了一种在理论上一致且表达力强的线性表示方法，以描述模型的计算。TensorLens具有理论基础，我们的实证验证显示，它比以前的注意力聚合方法产生了更丰富的表示。我们的实验表明，注意力张量可以作为开发可解释性和模型理解工具的强大基础。我们的代码作为补充材料附上。
英文摘要：
Attention matrices are fundamental to transformer research, supporting a broad range of applications including interpretability, visualization, manipulation, and distillation. Yet, most existing analyses focus on individual attention heads or layers, failing to account for the model's global behavior. While prior efforts have extended attention formulations across multiple heads via averaging and matrix multiplications or incorporated components such as normalization and FFNs, a unified and complete representation that encapsulates all transformer blocks is still lacking. We address this gap by introducing TensorLens, a novel formulation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction tensor. This tensor jointly encodes attention, FFNs, activations, normalizations, and residual connections, offering a theoretically coherent and expressive linear representation of the model's computation. TensorLens is theoretically grounded and our empirical validation shows that it yields richer representations than previous attention-aggregation methods. Our experiments demonstrate that the attention tensor can serve as a powerful foundation for developing tools aimed at interpretability and model understanding. Our code is attached as a supplementary.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
