TP-Spikformer: 令牌修剪的脉冲变换器
TP-Spikformer: Token Pruned Spiking Transformer
AI总结：
TP-Spikformer通过令牌修剪减少计算开销，保持性能，适用于资源受限的SNN部署。
摘要：
脉冲神经网络（SNNs）因其事件驱动的计算范式，为传统神经网络提供了一种节能的替代方案。然而，最近的脉冲变换器进展集中在通过大规模架构提高准确性，这需要大量计算资源，限制了在资源受限设备上的部署。在本文中，我们提出了一种简单而有效的令牌修剪方法，称为TP-Spikformer，用于脉冲变换器，以减少存储和计算开销，同时保持具有竞争力的性能。具体而言，我们首先引入了一个启发式时空信息保留准则，全面评估令牌的重要性，为保留信息令牌分配更高的分数，为修剪无信息令牌分配较低的分数。基于此准则，我们提出了一种信息保留令牌修剪框架，采用块级早期停止策略来处理无信息令牌，而不是直接移除它们。这也有助于在令牌修剪过程中保留更多信息。我们通过在不同架构上进行广泛实验来展示TP-Spikformer的有效性、效率和可扩展性，包括Spikformer、QKFormer和Spike-driven Transformer V1和V3，以及图像分类、目标检测、语义分割和基于事件的目标跟踪等任务。特别是，TP-Spikformer在无训练的情况下表现良好。这些结果揭示了其作为在有限计算资源下部署SNNs的高效且实用解决方案的潜力。
英文摘要：
Spiking neural networks (SNNs) offer an energy-efficient alternative to traditional neural networks due to their event-driven computing paradigm. However, recent advancements in spiking transformers have focused on improving accuracy with large-scale architectures, which require significant computational resources and limit deployment on resource-constrained devices. In this paper, we propose a simple yet effective token pruning method for spiking transformers, termed TP-Spikformer, that reduces storage and computational overhead while maintaining competitive performance. Specifically, we first introduce a heuristic spatiotemporal information-retaining criterion that comprehensively evaluates tokens' importance, assigning higher scores to informative tokens for retention and lower scores to uninformative ones for pruning. Based on this criterion, we propose an information-retaining token pruning framework that employs a block-level early stopping strategy for uninformative tokens, instead of removing them outright. This also helps preserve more information during token pruning. We demonstrate the effectiveness, efficiency and scalability of TP-Spikformer through extensive experiments across diverse architectures, including Spikformer, QKFormer and Spike-driven Transformer V1 and V3, and a range of tasks such as image classification, object detection, semantic segmentation and event-based object tracking. Particularly, TP-Spikformer performs well in a training-free manner. These results reveal its potential as an efficient and practical solution for deploying SNNs in real-world applications with limited computational resources.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
