序列 Token 合并：重新审视隐藏状态
Sequential Token Merging: Revisiting Hidden States
AI总结：
针对 Vision Mamba 的 token 二次扩展瓶颈，本文提出 STM，通过双向最近邻合并与 class token 隐藏状态保护，在大幅减少 token 的同时保持精度并揭示状态空间模型动力学。
摘要：
Vision Mambas（ViMs）以次二次复杂度取得了显著成功，但其效率仍受到 token 数量随图像分辨率呈二次增长的限制。现有方法虽然处理了 token 冗余问题，却忽视了 ViMs 内在的有限方向序列依赖（Limited Directional Sequential Dependence，LDSD）——我们的分析揭示，这是一种关键的信息流机制。我们进一步发现，Mamba 的选择性扫描能够在隐藏状态中实现渐进式信息聚合。基于这些洞察，我们提出序列 Token 合并（Sequential Token Merging，STM），其特点包括：1）双向最近邻合并，通过对称空间聚合保留序列依赖；2）隐藏状态保护，以稳定 class token 周围的隐藏状态。STM 策略性地利用 Mamba 的逐层损失收敛，将时间上的遗忘转化为稳定性。实验证明了 STM 的优越性：在减少 20% token 时，ViM-Ti 的准确率仅下降 1.0%；在减少 40% token 时，ViM-S 的性能仅退化 1.4%。我们的方法以极低的复杂度实现了最先进的效率，同时为状态空间模型动力学提供了新的见解。代码即将发布。
英文摘要：
Vision Mambas (ViMs) achieve remarkable success with sub-quadratic complexity, but their efficiency remains constrained by quadratic token scaling with image resolution. While existing methods address token redundancy, they overlook ViMs' intrinsic Limited Directional Sequential Dependence (LDSD) - a critical information flow mechanism revealed in our analysis. We further identify Mamba's selective scan enables gradual information aggregation in hidden states. Based on these insights, we propose Sequential Token Merging (STM), featuring: 1) Bidirectional nearest neighbor merging to preserve sequential dependencies through symmetric spatial aggregation, and 2) Hidden states protection to stabilize the hidden states around the class token. STM strategically leverages Mamba's layer-wise loss convergence to convert temporal forgetfulness into stability. Experiments demonstrate STM's superiority: 1.0% accuracy drop for ViM-Ti at 20% token reduction, and only 1.4% degradation for ViM-S at 40% reduction. Our method achieves state-of-the-art efficiency with minimal complexity, while providing new insights into state-space model dynamics. Codes will be released soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
