arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.21135cs.CRcs.AI
收藏

更强的评判者并未看到更多:LLM评估中的部署完整性盲区

Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation

University of Connecticut(康涅狄格大学)

Jianchang Su, Dawei Xiang, Yifan Zhang, Wei Zhang

AI总结:

该研究针对LLM评估中存在的部署完整性盲区问题,构建DIB-Bench基准开展实验,发现更强的评估器对篡改输出的接受率更高,提出了可零样本检测篡改的清单哈希身份通道及分级部署身份描述符。

摘要:

大语言模型智能体(LLM-agent)评估会报告行为分数,但未报告部署的运行时栈的身份。一个保留了评估输出的被篡改部署可以通过所有仅基于输出的评估器。我们引入DIB-Bench,这是一个基准,在7560个带分数示例、6种篡改类型以及从正则表达式评分器到LLM评判者再到GPT-5-nano的评估器阶梯上测量这种部署完整性盲区。更强的评估器接受更多被篡改的输出:平均接受率从88.6%(基于规则)上升到94.1%(GPT-5-nano),且在评估套件上不可见的触发条件篡改会产生48%-100%的有害输出,且随模型能力提升而上升。一个PAC风格的界表明,任何样本量下,输出无偏移的篡改都无法从输出中检测到;而清单哈希身份通道可在零样本下检测到它们。行为指纹识别能以高于随机水平(76.7%)识别模型组件,但仅覆盖6个栈组件中的1个。我们还为评估报告提出了分级部署身份描述符。

英文摘要:

LLM-agent evaluation reports behavioral scores but leaves the identity of the deployed runtime stack unreported. A tampered deployment that preserves evaluated outputs passes every output-only evaluator. We introduce DIB-Bench, a benchmark that measures this deployment-integrity blindness across 7,560 scored examples, six tamper types, and an evaluator ladder from regex scorers through LLM judges to GPT-5-nano. Stronger evaluators accept more tampered outputs: mean acceptance rises from 88.6% (rule-based) to 94.1% (GPT-5-nano), and a trigger-conditional tamper invisible on the evaluation suite produces 48-100% harmful outputs, rising with model capability. A PAC-style bound shows that tampers with zero output shift are undetectable from outputs at any sample size; a manifest-hash identity channel detects them with zero samples. Behavioral fingerprinting identifies the model component above chance (76.7%) but covers one of six stack components. We also propose a graded deployment-identity descriptor for evaluation reports.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑