发表机构
Bar-Ilan University（巴伊兰大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出TR-PTQ，通过泰勒区域重构和整数算术实现Transformer训练后量化，消除浮点硬件需求，在视觉和语言基准上精度损失小于1.5%。
AI 中文摘要
训练后量化（PTQ）能够实现高效部署，然而由于非线性层，Transformer架构仍然难以量化。虽然现有方法将精度损失归因于数值精度不足，常常需要浮点回退，但我们证明性能下降实际上是由特定的结构性误差源驱动的。我们发现，归一化层中学习到的缩放参数和GELU中的复合近似是主要的误差贡献者，而SoftMax对激进量化本质上保持鲁棒。为解决这些瓶颈，我们提出了TR-PTQ，一种使用共享泰勒区域（TR）指数和对数原语的统一整数专用公式。该方法允许计算昂贵的操作，包括除法和平方根，完全在对数域中通过标准整数算术执行。结合对LayerNorm参数的无校准、离群点感知优化，我们的方法消除了对非线性浮点硬件单元的需求，在视觉和语言基准上实现了小于1.5%的绝对精度下降。
英文摘要
Post-training quantization (PTQ) enables efficient deployment, yet transformer architectures remain challenging to quantize due to nonlinear layers. While existing methods attribute accuracy loss to insufficient numerical precision, often necessitating floating-point fallbacks, we demonstrate that degradation is actually driven by specific structural error sources. We find that learned scale parameters in normalization layers and compounded approximations in GELU are the primary error contributors, whereas SoftMax remains inherently robust to aggressive quantization. To address these bottlenecks, we introduce TR-PTQ, a unified integer-only formulation using shared Taylor Region (TR) exponential and logarithm primitives. This approach allows computationally expensive operations, including division and square roots, to be performed entirely in the log-domain via standard integer arithmetic. Combined with a calibration-free, outlier-aware optimization for LayerNorm parameters, our method eliminates the need for floating-point hardware units for nonlinearities, achieving less than 1.5\% absolute accuracy degradation across vision and language benchmarks.
