PKS⁴：并行运动选择状态空间扫描器用于高效的视频理解
$\text{PKS}^4$:Parallel Kinematic Selective State Space Scanners for Efficient Video Understanding
AI总结：
本文提出PKS⁴，通过并行扫描器保留空间结构，以线性复杂度实现高效的视频理解，显著降低训练计算量。
摘要：
时序建模仍然是视频理解中的基本挑战，尤其是在序列长度增加时。传统的依赖密集时空注意力的视频模型在长视频中面临二次计算成本。为克服这些成本，最近的方法通过参数高效微调（PEFT）方法如适配器将图像模型适应于视频。然而，深入插入这些模块在反向传播过程中会带来可接受的激活内存开销。尽管最近的高效状态空间模型（SSMs）引入了线性复杂度，但它们破坏了二维空间关系并依赖于广泛的掩码预训练来恢复空间意识。为克服这些限制，我们提出了并行运动选择状态空间扫描器（PKS⁴）。我们保留了一个标准的二维视觉骨干网络以保留空间语义，并插入一个具有线性复杂度时序扫描的单个插拔式PKS⁴模块，避免了时间注意力和多层适配器。我们首先通过运动先验编码器提取运动先验，通过帧间相关性和差异性捕获局部位移和运动边界。这些先验驱动线性复杂度的SSMs来跟踪底层运动状态，自适应地调节每个时间步的更新速度和读写策略。与全局扫描不同，我们沿时间维度为每个空间位置部署并行扫描器，保留空间结构的同时减少开销。在空间密集和时间密集的动作识别基准测试中，PKS⁴实现了最先进的性能。值得注意的是，我们的方法在仅仅20个epoch内收敛，训练计算量约为纯视频SSMs的10倍，建立了高效视频理解的新范式。
英文摘要：
Temporal modeling remains a fundamental challenge in video understanding, particularly as sequence lengths scale. Traditional video models relying on dense spatiotemporal attention suffer from quadratic computational costs for long videos. To circumvent these costs, recent approaches adapt image models for videos via Parameter-Efficient Fine-Tuning (PEFT) methods such as adapters. However, deeply inserting these modules incurs prohibitive activation memory overhead during back-propagation. While recent efficient State Space Models (SSMs) introduce linear complexity, they disrupt 2D spatial relationships and rely on extensive masked pre-training to recover spatial awareness. To overcome these limitations, we propose Parallel Kinematic Selective State Space Scanners (PKS$^4$). We retain a standard 2D vision backbone for spatial semantics and insert a single plug-and-play PKS$^4$ module with linear-complexity temporal scanning, avoiding temporal attention and multi-layer adapters. We first extract kinematic priors via a Kinematic Prior Encoder, which captures local displacements and motion boundaries through inter-frame correlations and differences. These priors drive linear-complexity SSMs to track underlying kinematic states, adaptively modulating update speeds and read-write strategies at each time step. Instead of global scanning, we deploy parallel scanners along the temporal dimension for each spatial location, preserving spatial structures while reducing overhead. Experiments on spatial-heavy and temporal-heavy action recognition benchmarks show that PKS$^4$ achieves state-of-the-art performance. Remarkably, our method converges in merely $20$ epochs, achieving approximately $10\times$ lower training compute than pure video SSMs, establishing a new paradigm for efficient video understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
