每比特都重要：量化转换器中精度-表达力权衡的理论研究
Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers
AI总结：
该研究揭示了量化转换器中精度与表达力之间的理论权衡，指出等式比较任务对量化敏感，需根据具体任务选择合适精度。
摘要：
量化减少了Transformer计算的数值精度，并被广泛用于加速推理，但其对表达力的影响仍缺乏充分描述。我们展示了表达力与精度之间精细的理论权衡：对于每一个p，我们展示了一个受等式函数启发的函数Γ，并证明一个单层softmax Transformer可以在p位精度下计算Γ，但不能在p-1位精度下计算。这一结果具体解释了在使用量化时广泛观察到的经验性表达力损失现象。实际上，它表明需要等式类比的比较（精确匹配、成员关系等）的任务对量化特别敏感。即使减少一位，也可能跨越一个阈值，使得模型无法可靠地表示所需的比较。因此，它为开发有助于从业者选择可能的量化程度的启发式方法铺平了道路：精度应根据要检查的等式长度来选择特定任务。我们的证明结合了显式的有限精度Transformer构造与通信复杂度下界，得出一个紧致的
英文摘要：
Quantization reduces the numerical precision of Transformer computations and is widely used to accelerate inference, yet its effect on expressivity remains poorly characterized. We demonstrate a fine-grained theoretical tradeoff between expressivity and precision: For every p we exhibit a function Γ, inspired by the equality function, and prove that a one-layer softmax Transformer can compute Γ, with p bits of precision, but not with p-1 bits of precision. This result concretely explains the widely observed phenomenon of empirical loss of expressivity when quantization is used. Practically, it suggests that tasks requiring equality-like comparisons (exact match, membership, etc.) are especially sensitive to quantization. Dropping even one bit can cross a threshold where the model cannot represent the needed comparison reliably. Thus, it paves the way for developing heuristics that will help practitioners choose how much quantization is possible: the precision should be chosen as a function of the length of equality to be checked for the specific task. Our proofs combine explicit finite-precision Transformer constructions with communication-complexity lower bounds, yielding a tight "one-bit" threshold.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
