arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2007.08442cs.CVcs.LG
收藏

克罗内克注意力网络

Kronecker Attention Networks

Texas A&M University(德克萨斯农工大学)

Hongyang Gao, Zhengyang Wang, Shuiwang Ji

摘要:

注意力算子已经应用于文本等一维数据和图像和视频等高阶数据。在高阶数据上使用注意力操作符需要将空间或时空维度平坦化为向量,该向量被假设为遵循多元正态分布。这不仅会导致对计算资源的过度要求,而且无法保留数据中的结构。在这项工作中,我们建议通过假设数据遵循矩阵变量正态分布来避免扁平化。基于这种新的观点,我们开发了直接对高阶张量数据进行操作的Kronecker注意力算子(KAOs)。更重要的是,拟议的KAO导致计算资源的显着减少。实验结果表明,我们的方法将所需的计算资源量减少了数百倍,对于高维和高阶数据具有更大的因素。研究结果还表明,具有KAO的网络优于没有注意力的模型,同时与具有原始注意力算子的网络相比,具有竞争力的性能。

英文摘要:

Attention operators have been applied on both 1-D data like texts and higher-order data such as images and videos. Use of attention operators on high-order data requires flattening of the spatial or spatial-temporal dimensions into a vector, which is assumed to follow a multivariate normal distribution. This not only incurs excessive requirements on computational resources, but also fails to preserve structures in data. In this work, we propose to avoid flattening by assuming the data follow matrix-variate normal distributions. Based on this new view, we develop Kronecker attention operators (KAOs) that operate on high-order tensor data directly. More importantly, the proposed KAOs lead to dramatic reductions in computational resources. Experimental results show that our methods reduce the amount of required computational resources by a factor of hundreds, with larger factors for higher-dimensional and higher-order data. Results also show that networks with KAOs outperform models without attention, while achieving competitive performance as those with original attention operators.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑