arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.04515cs.DC
收藏

QuantPipe:将自适应后训练Quantization应用于动态边缘环境中的分布式Transformer管道

QuantPipe: Applying Adaptive Post-Training Quantization for Distributed Transformer Pipelines in Dynamic Edge Environments

Haonan Wang, Connor Imes, Souvik Kundu, Peter A. Beerel, Stephen P. Crago, John Paul Walters

摘要:

流水线并行在云环境中部署大规模Transformer模型方面取得了巨大成功,但在边缘环境中受到的关注较少。与具有高速和稳定网络互连的云场景不同,边缘系统中的动态带宽可能会降低分布式管道性能。我们使用QuantPipe解决这个问题,QuantPipe是一种通信高效的分布式边缘系统,它引入了训练后量化(PTQ)来压缩通信张量。QuantPipe使用自适应PTQ来更改位宽以响应带宽动态,从而保持Transformer管道性能,同时导致有限的推理精度损失。我们进一步提高了精度与直接搜索分析裁剪整数量化方法(DS-ACIQ),估计和真实数据分布之间的差距的桥梁。实验结果表明,QuantPipe适应动态带宽以保持流水线性能,同时使用宽范围的量化位宽实现实际的模型精度,例如,在ImageNet上,与朴素量化相比,2位量化下的准确率提高了15.85%。

英文摘要:

Pipeline parallelism has achieved great success in deploying large-scale transformer models in cloud environments, but has received less attention in edge environments. Unlike in cloud scenarios with high-speed and stable network interconnects, dynamic bandwidth in edge systems can degrade distributed pipeline performance. We address this issue with QuantPipe, a communication-efficient distributed edge system that introduces post-training quantization (PTQ) to compress the communicated tensors. QuantPipe uses adaptive PTQ to change bitwidths in response to bandwidth dynamics, maintaining transformer pipeline performance while incurring limited inference accuracy loss. We further improve the accuracy with a directed-search analytical clipping for integer quantization method (DS-ACIQ), which bridges the gap between estimated and real data distributions. Experimental results show that QuantPipe adapts to dynamic bandwidth to maintain pipeline performance while achieving a practical model accuracy using a wide range of quantization bitwidths, e.g., improving accuracy under 2-bit quantization by 15.85\% on ImageNet compared to naive quantization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑