arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.02761cs.CVcs.AIcs.LGcs.MM
收藏

面向多模态任务的多层可学习注意力掩码

Multi-layer Learnable Attention Mask for Multimodal Tasks

Dartmouth College(达特茅斯学院)

Wayner Barrios, SouYoung Jin

AI总结:

针对Transformer自注意力机制在多模态等复杂场景下因token粒度差异和长序列计算量大导致效果不佳的问题,提出多层可学习注意力掩码(LAM)以全局调节注意力图、优先关键token,在缓解冗余计算的同时显著提升模型性能。

摘要:

尽管 Transformer 模型中的自注意力机制在许多领域已被证明有效,但由于每个 token 的粒度不同以及长序列的高计算需求,我们观察到它在更多样化的设置(例如多模态)中效果较差。为了解决这些挑战,我们引入了可学习注意力掩码(LAM),其策略性设计旨在全局调节注意力图并优先处理序列中的关键 token。利用类 BERT transformer 网络中的自注意力模块,我们的方法熟练地捕获了 token 之间的关联。将 LAM 扩展到多层版本,以适应 Transformer 网络每一层中嵌入的多样信息方面。在 MADv2、QVHighlights、ImageNet 1K 和 MSRVTT 等各种数据集上的全面实验验证,证明了 LAM 的有效性,展示了其在缓解冗余计算的同时增强模型性能的能力。这种开创性的方法在增强复杂场景(例如电影理解)的理解方面取得了重大进展。

英文摘要:

While the Self-Attention mechanism in the Transformer model has proven to be effective in many domains, we observe that it is less effective in more diverse settings (e.g. multimodality) due to the varying granularity of each token and the high computational demands of lengthy sequences. To address the challenges, we introduce the Learnable Attention Mask (LAM), strategically designed to globally regulate attention maps and prioritize critical tokens within the sequence. Leveraging the Self-Attention module in a BERT-like transformer network, our approach adeptly captures associations between tokens. The extension of the LAM to a multi-layer version accommodates the varied information aspects embedded at each layer of the Transformer network. Comprehensive experimental validation on various datasets, such as MADv2, QVHighlights, ImageNet 1K, and MSRVTT, demonstrates the efficacy of the LAM, exemplifying its ability to enhance model performance while mitigating redundant computations. This pioneering approach presents a significant advancement in enhancing the understanding of complex scenarios, such as in movie understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑