MaMe & MaRe：基于矩阵的令牌合并与恢复用于高效的视觉感知与合成
MaMe & MaRe: Matrix-Based Token Merging and Restoration for Efficient Visual Perception and Synthesis
AI总结：
本文提出MaMe和MaRe，通过矩阵运算实现免训练的令牌合并与恢复，提升视觉模型效率，实验显示在ViT-B上吞吐量提升2%且精度下降1.0%，在图像合成中减少Stable Diffusion v2.1生成延迟31%。
摘要：
令牌压缩对于缓解视觉Transformer（ViTs）中自注意力机制的二次复杂度至关重要，尤其是当输入令牌数量庞大时。现有方法如ToMe依赖于GPU效率低的操作（例如排序、散列写入），引入了限制其有效性的开销。我们引入MaMe，一种基于完全矩阵运算的免训练、可微分的令牌合并方法，具有GPU友好性以加速ViTs。此外，我们提出了MaRe，其逆操作，用于令牌恢复，形成MaMe+MaRe流水线用于图像合成。当应用于预训练模型时，MaMe使ViT-B的吞吐量翻倍，精度下降2%。值得注意的是，通过MaMe微调最后一层可使ViT-B精度提升1.0%并加快1.1倍。在SigLIP2-B@512零样本分类中，MaMe提供1.3倍加速，性能损失可忽略不计。在视频任务中，MaMe在Kinetics-400上将VideoMAE-L加速48.5%，仅损失0.84%的精度。此外，MaMe在某些任务中同时提升了性能和速度。在图像合成中，MaMe+MaRe流水线在提升质量的同时，将Stable Diffusion v2.1生成延迟减少了31%。这些结果证明了MaMe和MaRe在加速视觉模型方面的有效性。代码可在https://github.com/cominder/mame获取。
英文摘要：
Token compression is crucial for mitigating the quadratic complexity of self-attention mechanisms in Vision Transformers (ViTs), which often involve numerous input tokens. Existing methods, such as ToMe, rely on GPU-inefficient operations (e.g., sorting, scattered writes), introducing overheads that limit their effectiveness. We introduce MaMe, a training-free, differentiable token merging method based entirely on matrix operations, which is GPU-friendly to accelerate ViTs. Additionally, we present MaRe, its inverse operation, for token restoration, forming a MaMe+MaRe pipeline for image synthesis. When applied to pre-trained models, MaMe doubles ViT-B throughput with a 2% accuracy drop. Notably, fine-tuning the last layer with MaMe boosts ViT-B accuracy by 1.0% at 1.1x speed. In SigLIP2-B@512 zero-shot classification, MaMe provides 1.3x acceleration with negligible performance degradation. In video tasks, MaMe accelerates VideoMAE-L by 48.5% on Kinetics-400 with only a 0.84% accuracy loss. Furthermore, MaMe achieves simultaneous improvements in both performance and speed on some tasks. In image synthesis, the MaMe+MaRe pipeline enhances quality while reducing Stable Diffusion v2.1 generation latency by 31%. Collectively, these results demonstrate MaMe's and MaRe's effectiveness in accelerating vision models. The code is available at https://github.com/cominder/mame}{https://github.com/cominder/mame.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
