arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.19790cs.AIcs.LG
收藏

大语言模型中的隐藏可靠性风险:系统性识别精度诱导的输出分歧

Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements

Shanghai Jiao Tong University(上海交通大学);Beihang University(北航);Nanyang Technological University(南洋理工大学);Singapore Management University(新加坡管理学院)

Yifei Wang, Tianlin Li, Xiaohan Zhang, Xiaoyu Zhang, Wei Ma, Mingfei Cheng, Li Pan

AI总结:

本文提出PrecisionDiff框架,用于系统检测大语言模型在不同精度下的行为分歧,通过生成精度敏感测试输入并进行跨精度比较分析,揭示传统方法难以发现的细微差异,实验表明此类分歧在多个开源对齐LLM中普遍存在,PrecisionDiff在检测此类问题上表现优异。

摘要:

大语言模型(LLMs)越来越多地在不同的数值精度配置下部署,包括标准浮点格式(如bfloat16和float16)和量化整数格式(如int16和int8),以满足效率和资源约束。然而,不同精度的LLMs之间的微小不一致难以检测,现有评估方法往往忽视这些不一致。本文提出PrecisionDiff,一种自动化差分测试框架,用于系统性地检测LLMs中的精度诱导行为分歧。PrecisionDiff生成对精度敏感的测试输入,并进行跨精度比较分析,以揭示传统测试策略下隐藏的细微差异。为了证明其实际意义,我们将其应用于对齐验证任务,其中精度诱导的分歧表现为在一种精度下被拒绝的输入在另一种精度下可能产生有害响应。实验结果表明,此类行为分歧在多个开源对齐LLM和精度设置中普遍存在,且PrecisionDiff在检测这些问题上显著优于传统测试方法。我们的工作实现了自动化精度敏感测试生成,有助于有效的预部署评估,并在训练过程中提高精度鲁棒性。

英文摘要:

Large language models (LLMs) are increasingly deployed under diverse numerical precision configurations, including standard floating-point formats (e.g., bfloat16 and float16) and quantized integer formats (e.g., int16 and int8), to meet efficiency and resource constraints. However, minor inconsistencies between LLMs of different precisions are difficult to detect and are often overlooked by existing evaluation methods. In this paper, we present PrecisionDiff, an automated differential testing framework for systematically detecting precision-induced behavioral disagreements in LLMs. PrecisionDiff generates precision-sensitive test inputs and performs cross-precision comparative analysis to uncover subtle divergences that remain hidden under conventional testing strategies. To demonstrate its practical significance, we instantiate PrecisionDiff on the alignment verification task, where precision-induced disagreements manifest as jailbreak divergence-inputs that are rejected under one precision may produce harmful responses under another. Experimental results show that such behavioral disagreements are widespread across multiple open-source aligned LLMs and precision settings, and that PrecisionDiff significantly outperforms vanilla testing methods in detecting these issues. Our work enables automated precision-sensitive test generation, facilitating effective pre-deployment evaluation and improving precision robustness during training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑