硅 bureaucracy 与 AI 考试导向教育：LLM 测试基准中的污染敏感性与分数可信度
Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks
AI总结：
本文探讨了基于测试基准的LLM评估体系，指出其依赖于基准分数直接反映真实泛化能力的假设，并提出审计框架以分析污染敏感性和分数可信度。
摘要：
公共基准越来越多地决定大型语言模型（LLM）的排名、选择和部署方式。我们将这一以基准为中心的制度称为硅 bureaucracy 与 AI 考试导向教育，并认为其基于一个脆弱的假设：即基准分数直接反映真实的泛化能力。然而，在实践中，这些分数可能将考试导向的能力与原则性能力混淆，尤其是在现代训练流程中难以排除污染和语义泄漏的情况下。因此，我们提出一个审计框架，用于分析LLM基准中的污染敏感性和分数可信度。使用路由器-工人设置，我们比较了清洁对照条件与噪声条件，其中基准问题被系统性地删除、重写和扰动后再传递下游。对于真正干净的基准，噪声条件不应一致优于清洁对照基线。然而，在多个模型中，我们发现噪声条件下普遍存在但异质的基线以上收益，表明基准相关线索可能被重新组装，并可以激活与污染相关的记忆。这些结果表明，相似的基准分数可能具有截然不同的可信度水平。与其完全拒绝基准，我们主张基于基准的评估应补充明确的污染敏感性和分数可信度审计。
英文摘要：
Public benchmarks increasingly govern how large language models (LLMs) are ranked, selected, and deployed. We frame this benchmark-centered regime as Silicon Bureaucracy and AI Test-Oriented Education, and argue that it rests on a fragile assumption: that benchmark scores directly reflect genuine generalization. In practice, however, such scores may conflate exam-oriented competence with principled capability, especially when contamination and semantic leakage are difficult to exclude from modern training pipelines. We therefore propose an audit framework for analyzing contamination sensitivity and score confidence in LLM benchmarks. Using a router-worker setup, we compare a clean-control condition with noisy conditions in which benchmark problems are systematically deleted, rewritten, and perturbed before being passed downstream. For a genuinely clean benchmark, noisy conditions should not consistently outperform the clean-control baseline. Yet across multiple models, we find widespread but heterogeneous above-baseline gains under noisy conditions, indicating that benchmark-related cues may be reassembled and can reactivate contamination-related memory. These results suggest that similar benchmark scores may carry substantially different levels of confidence. Rather than rejecting benchmarks altogether, we argue that benchmark-based evaluation should be supplemented with explicit audits of contamination sensitivity and score confidence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
