LLM-agent evaluation reports behavioral scores but leaves the identity of the deployed runtime stack unreported. A tampered deployment that preserves evaluated outputs passes every output-only evaluator. We introduce DIB-Bench, a benchmark that measures this deployment-integrity blindness across 7,560 scored examples, six tamper types, and an evaluator ladder from regex scorers through LLM judges to GPT-5-nano. Stronger evaluators accept more tampered outputs: mean acceptance rises from 88.6% (rule-based) to 94.1% (GPT-5-nano), and a trigger-conditional tamper invisible on the evaluation suite produces 48-100% harmful outputs, rising with model capability. A PAC-style bound shows that tampers with zero output shift are undetectable from outputs at any sample size; a manifest-hash identity channel detects them with zero samples. Behavioral fingerprinting identifies the model component above chance (76.7%) but covers one of six stack components. We also propose a graded deployment-identity descriptor for evaluation reports.
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA(卡内基梅隆大学机器学习系);Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA(加州大学圣地亚哥分校计算机科学与工程系);Department of Computer Science, University of Virginia, Charlottesville, VA, USA(弗吉尼亚大学计算机科学系);Martian Research, San Francisco, California, USA(火星研究公司);Apart Research, San Francisco, California, USA(Apart研究公司)