arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2506.09501cs.CL
收藏

理解并缓解LLM推理中数值不确定性的来源

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference

Rice University(Rice大学);University of Minnesota Twin Cities(明尼苏达大学双城分校);Adobe Inc.(Adobe公司)

Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, Zirui Liu

AI总结:

本研究揭示LLM推理中因浮点非结合性导致的数值不确定性,并开发LayerCast流水线,以FP32计算与16位存储平衡精度与稳定性,提升可复现性。

摘要:

大型语言模型(LLMs)现已广泛应用于各个领域,并展现出令人瞩目的性能。然而,这一进展基于一个前提,即基准测试分数既准确又可复现。我们证明,LLM性能的可复现性是脆弱的:改变系统配置,如评估批次大小、GPU数量和GPU版本,可能导致生成响应出现显著差异。这一问题在推理模型中尤为突出,早期token的微小舍入差异可能级联成不同的思维链,最终影响准确性。例如,在bfloat16精度和贪婪解码下,像DeepSeek-R1-Distill-Qwen-7B这样的推理模型,由于GPU数量、类型和评估批次大小的不同,其准确率差异可达9%,响应长度差异可达9000个token。我们将这种变异性的根本原因追溯到有限数值精度下浮点运算的非结合性。这项工作首次系统研究了数值精度如何影响LLM推理中的可复现性。通过在多种硬件、软件和精度设置下进行精心控制的实验,我们量化了模型输出何时以及如何发生分歧。我们的分析揭示,浮点精度——虽然对可复现性至关重要——在评估实践中常常被忽视。受此启发,我们开发了一个轻量级推理流水线,名为LayerCast,它以16位精度存储权重,但所有计算均在FP32中执行,从而在内存效率与数值稳定性之间取得平衡。代码可在https://github.com/nanomaoli/llm_reproducibility获取。

英文摘要:

Large Language Models (LLMs) are now integral across various domains and have demonstrated impressive performance. Progress, however, rests on the premise that benchmark scores are both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. Our analysis reveals that floating-point precision - while critical for reproducibility - is often neglected in evaluation practices. Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at https://github.com/nanomaoli/llm_reproducibility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑