超越准确率：低精度快速矩阵乘法的前缀不变实现

中文摘要 AI 辅助

快速矩阵乘法通过精确相消来节省乘法运算，但舍入那些混合了token行的求和可能会使后续token的贡献出现在较早的语言模型输出中。这威胁到了前缀不变性，而多项选择似然评分正依赖于这一性质：被评分的似然必须仅依赖于其允许的前缀。在Qwen2.5-14B-Instruct上，两种修复到普通外观准确率的快速FP8实现，当仅将允许前缀之后的文本替换为bf16模型自身的贪心续接时，在240个OpenBookQA题目中分别有5.83%和10.00%的题目改变了似然所选择的答案。两种行局部控制方法，即bf16模型和部署的FP8矩阵乘法内核，均未改变任何答案。因此，准确率并不能证明前缀不变性，而我们分析的稳定性判据也无法区分这些实现：在两级Strassen的所有512个符号变体中，这些判据保持不变，而教师强制困惑度在同一模型上跨越了772.4倍的范围。因此，我们构造了有界整数编码上两级Strassen的认证实现，该实现独立量化token行，然后在重新缩放之前精确地进行混合和相消，使用49次块乘法而非64次。我们的证书保证了在相同量化规格下与指定的行局部经典int8算子逐位相等，因此每个认证实现都继承了其前缀不变性。认证因此将实现选择转变为纯粹的成本决策：运行哪个认证实现不再能改变任何一个被评分的似然。

英文摘要

Fast matrix multiplication saves multiplications through exact cancellation, but rounding sums that mix token rows can leave contributions from later tokens in earlier language model outputs. This threatens prefix invariance, which multiple-choice likelihood scoring relies on: a scored likelihood must depend only on its allowed prefix. On Qwen2.5-14B-Instruct, two fast FP8 realizations repaired to ordinary-looking accuracy still change the answers chosen by likelihood on 5.83% and 10.00% of 240 OpenBookQA items when only the text after the allowed prefix is replaced with the bf16 model's own greedy continuation. Both row-local controls, the bf16 model and a deployed FP8 matrix multiplication kernel, change none. Accuracy thus does not certify prefix invariance, and the stability criteria we analyze cannot tell realizations apart: across all 512 sign variants of two-level Strassen they stay constant while teacher-forced perplexities span a 772.4$\times$ range on the same model. We therefore construct certified realizations of two-level Strassen on bounded integer codes that quantize token rows independently, then mix and cancel exactly before rescaling, using 49 block multiplications instead of 64. Our certificate guarantees bitwise equality to a prescribed row-local classical int8 operator at the same quantization specification, so every certified realization inherits its prefix invariance. Certification thus turns realization choice into a pure cost decision: which certified realization runs can no longer change a single scored likelihood.

发表机构

• Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC)（北京通明湖信息技术应用创新中心）
• Fudan University Institute of Systems for Advanced Computing（复旦大学先进计算系统研究院）
• Harbin Institute of Technology（哈尔滨工业大学）
• Key Lab of HCST (PKU), MOE（北京大学高可信软件技术教育部重点实验室；北京大学计算机学院）
• SCS, Peking University（上海开放计算系统研究院）
• Shanghai Institute of Systems for Open Computing

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

23 pages, 4 figures