arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.28708cs.LGcs.DC
收藏

基于GPU的Transformer神经网络优化用于实时推断

GPU-Accelerated Optimization of Transformer-Based Neural Networks for Real-Time Inference

Soutrik Mukherjee, Sangwhan Cha

AI总结:

本文提出了一种基于NVIDIA TensorRT的GPU加速推理管道,通过混合精度优化提升BERT和GPT-2的性能,实现高达64.4倍的速度提升和内存使用减少63%。

摘要:

本文提出了基于NVIDIA TensorRT的GPU加速推理管道的设计与评估,采用混合精度优化技术对Transformer模型进行优化。我们评估了BERT-base(110M参数)和GPT-2(124M参数)在不同批量大小和序列长度下的性能。该系统在CPU基准上实现了高达64.4倍的速度提升,单样本推理延迟低于10毫秒,并将内存使用减少了63%。我们引入了一种混合精度策略,将FP32用于数值敏感的操作如softmax和层归一化,而将FP16应用于线性层。这种方法保持了高数值精度(与基准输出的余弦相似度≥0.9998)并消除了NaN不稳定性。该管道实现为模块化、容器化的系统,支持在超过360种配置中进行可重复的基准测试。跨GPU验证显示,NVIDIA A100上的FP16速度提升比率为1.84x至2.00x,且具有稳定的数值行为。在SST-2下游评估中,混合精度未导致准确性下降。在WikiText-2验证中,随机输入在完全FP16下对NaN不稳定性低估达6倍,同时确认了混合方法的鲁棒性(0.0%的NaN,余弦相似度≥0.9998)。这些结果提供了在GPU架构上性能和准确性权衡的详细表征,并为在延迟敏感环境中部署Transformer模型提供了实用指导。

英文摘要:

This paper presents the design and evaluation of a GPU-accelerated inference pipeline for transformer models using NVIDIA TensorRT with mixed-precision optimization. We evaluate BERT-base (110M parameters) and GPT-2 (124M parameters) across batch sizes from 1 to 32 and sequence lengths from 32 to 512. The system achieves up to 64.4x speedup over CPU baselines, sub-10 ms latency for single-sample inference, and a 63 percent reduction in memory usage. We introduce a hybrid precision strategy that preserves FP32 for numerically sensitive operations such as softmax and layer normalization, while applying FP16 to linear layers. This approach maintains high numerical fidelity (cosine similarity >= 0.9998 relative to baseline outputs) and eliminates NaN instability. The pipeline is implemented as a modular, containerized system that enables reproducible benchmarking across more than 360 configurations. Cross-GPU validation on an NVIDIA A100 shows consistent FP16 speedup ratios between 1.84x and 2.00x, along with stable numerical behavior. Downstream evaluation on SST-2 demonstrates no accuracy degradation under hybrid precision. Validation on WikiText-2 shows that random inputs underestimate NaN instability by up to 6x for full FP16, while confirming the robustness of the hybrid approach (0.0 percent NaN, cosine similarity >= 0.9998). These results provide a detailed characterization of performance and accuracy trade-offs across GPU architectures and offer practical guidance for deploying transformer models in latency-critical environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑