MeMSVD:利用增量SVD捕获长程时间结构
MeMSVD: Long-Range Temporal Structure Capturing Using Incremental SVD
AI总结:
本文提出MeMSVD方法,利用增量SVD对长时视频记忆库进行低秩近似,以替代注意力机制,在显著降低计算复杂度和内存占用的同时,达到或超越现有方法在三个数据集上的准确率。
摘要:
本文研究长时视频理解,其目标是在长达数分钟的长时间窗口内识别人体动作。在先前工作中,长时上下文通过构建一个由过去和未来视频特征组成的长时记忆库来捕获,然后利用注意力机制将其整合到标准(短时)视频识别骨干网络中。与该方法相关的两个众所周知的问题是注意力操作的二次复杂度,以及推理时必须将整个特征库存储在内存中。为了解决这两个问题,我们提出了一种基于注意力机制的替代方案,该方案基于使用奇异值分解(Singular Value Decomposition, SVD)获得的记忆低秩近似。我们的方案有两个优点:(a)它将复杂度降低了一个数量级以上;(b)它适合以增量方式高效计算记忆基,无需将整个特征库存储在内存中。所提出的方案在保持内存高效的同时,达到或超过了基于注意力机制所取得的准确率。通过大量实验,我们证明了我们的框架能够泛化到不同的架构和任务,在三个数据集上超越了当前最先进的方法。
英文摘要:
This paper is on long-term video understanding where the goal is to recognise human actions over long temporal windows (up to minutes long). In prior work, long temporal context is captured by constructing a long-term memory bank consisting of past and future video features which are then integrated into standard (short-term) video recognition backbones through the use of attention mechanisms. Two well-known problems related to this approach are the quadratic complexity of the attention operation and the fact that the whole feature bank must be stored in memory for inference. To address both issues, we propose an alternative to attention-based schemes which is based on a low-rank approximation of the memory obtained using Singular Value Decomposition. Our scheme has two advantages: (a) it reduces complexity by more than an order of magnitude, and (b) it is amenable to an efficient implementation for the calculation of the memory bases in an incremental fashion which does not require the storage of the whole feature bank in memory. The proposed scheme matches or surpasses the accuracy achieved by attention-based mechanisms while being memory-efficient. Through extensive experiments, we demonstrate that our framework generalises to different architectures and tasks, outperforming the state-of-the-art in three datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。