Large Language Models (LLMs) are now integral across various domains and have demonstrated impressive performance. Progress, however, rests on the premise that benchmark scores are both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. Our analysis reveals that floating-point precision - while critical for reproducibility - is often neglected in evaluation practices. Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at https://github.com/nanomaoli/llm_reproducibility.
Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(信息处理国家重点实验室,计算机科学学院,北京大学);University of Electronic Science and Technology of China(电子科技大学);Nanbeige Lab, BOSS Zhipin(纳贝格实验室,BOSS智联)
Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference
State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Huawei Technology(华为技术);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院)
Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系)
Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibration
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);CSIRO(澳大利亚联邦科学与工业研究组织);Center of Excellence for Generative AI, King Abdullah University of Science and Technology(阿卜杜拉国王科技大学生成式人工智能卓越中心)
MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai AI Lab, Shanghai Innovation Institute(上海人工智能实验室,上海创新研究院)
Detecting and Mitigating Treatment Leakage in Text-Based Causal Inference: Distillation and Sensitivity Analysis
Institute for Analytical Sociology, Linköping University, Sweden(分析社会学研究所,利乌波尔大学);Department of Computer Science and Engineering, Chalmers University of Technology, Sweden(计算机科学与工程系,查尔姆斯理工大学);Department of Government, University of Texas at Austin, USA(政府系,德克萨斯大学奥斯汀分校)