Q-S5:迈向量化状态空间模型
Q-S5: Towards Quantized State Space Models
AI总结:
本文通过 QAT 与 PTQ 系统量化 S5,评估其在动态系统、sMNIST 和多数 LRA 任务上的敏感性,实现准确率下降不足 1% 的全量化模型。
摘要:
在探索下一代序列建模架构的过程中,状态空间模型(SSM)已成为 transformers 的有力替代方案,尤其是因其计算效率以及对动态系统的适用性。本文研究量化对 S5 模型的影响,以理解其对模型性能的作用,并推动 S5 部署到边缘和资源受限平台。我们利用量化感知训练(QAT)和训练后量化(PTQ),系统评估 SSM 在不同任务上的量化敏感性,包括动态系统建模、Sequential MNIST(sMNIST)以及 Long Range Arena(LRA)中的大多数任务。我们提出全量化 S5 模型,其在 sMNIST 和大多数 LRA 任务上的测试准确率下降不到 1%。我们发现,当循环权重低于 8 位精度时,大多数任务的性能会显著下降,但其他组件可以进一步压缩而不会造成显著性能损失。结果还表明,PTQ 仅在基于语言的 LRA 任务上表现良好,而所有其他任务都需要 QAT。本研究为持续开发高效且硬件优化的 SSM 提供了必要见解。
英文摘要:
In the quest for next-generation sequence modeling architectures, State Space Models (SSMs) have emerged as a potent alternative to transformers, particularly for their computational efficiency and suitability for dynamical systems. This paper investigates the effect of quantization on the S5 model to understand its impact on model performance and to facilitate its deployment to edge and resource-constrained platforms. Using quantization-aware training (QAT) and post-training quantization (PTQ), we systematically evaluate the quantization sensitivity of SSMs across different tasks like dynamical systems modeling, Sequential MNIST (sMNIST) and most of the Long Range Arena (LRA). We present fully quantized S5 models whose test accuracy drops less than 1% on sMNIST and most of the LRA. We find that performance on most tasks degrades significantly for recurrent weights below 8-bit precision, but that other components can be compressed further without significant loss of performance. Our results further show that PTQ only performs well on language-based LRA tasks whereas all others require QAT. Our investigation provides necessary insights for the continued development of efficient and hardware-optimized SSMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。