LLM 是否知道自己正在被测试？GPT-OSS-20B 中的评估意识与激励敏感型失败
Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B
AI总结：
本研究通过六个A/B场景测试GPT-OSS-20B，发现评估导向的提示框架会虚增思维链长度和格式合规性，但准确率提升有限，且激励措辞和多语言标题影响错误构成，提示基准测试需反映真实部署能力。
摘要：
大型语言模型（LLM）的基准测试通常依赖于带有评分提示的提示词，这些提示词要求可见的推理过程和严格的格式，而实际部署则要求简洁、受合同约束的答案。我们研究这种“评估气味”是否会在没有相应能力提升的情况下虚增测量性能。使用单个开放权重模型（GPT-OSS-20B），我们运行了六个配对的 A/B 场景，这些场景保持任务内容和解码固定，同时改变框架（评估导向 vs. 现实世界）和推理深度（中/高）：确定性数学、严格代码修复、引文生成、激励翻转（谨慎 vs. 能力）、思维链（CoT）可见性以及多语言（乌尔都语）标题。确定性验证器计算准确率、仅答案合规性、回避/拒绝、思维链长度和模式合规性，并带有预注册的差异和综合指数。在所有场景中，评估框架可靠地增加了思维链长度（数百到超过1000个字符）并降低了仅答案合规性，而准确率提升有限或不一致。在结构化输出中，它改善了包装（例如，围栏块、枚举列表），但未改善经正则表达式验证的实质内容。激励措辞重新调整了错误构成：赞扬谨慎在高推理深度下适度提高准确率并减少错误但自信的错误，而赞扬能力则产生更简洁但风险更高的输出。乌尔都语评分标题重现了这些特征，并可能在更高推理深度下降低准确率，表明多语言一致性存在风险。我们提供了一个可复现的 A/B 框架（提示库、验证器、每次运行的分数、脚本；带版本 DOI）和实用指南：中性措辞或双框架检查、合同感知评分、风格差异报告、置信度治理以及多语言仪表板，以确保基准测试收益反映可部署的能力。
英文摘要：
Benchmarks for large language models (LLMs) often rely on rubric-scented prompts that request visible reasoning and strict formatting, whereas real deployments demand terse, contract-bound answers. We investigate whether such "evaluation scent" inflates measured performance without commensurate capability gains. Using a single open-weights model (GPT-OSS-20B), we run six paired A/B scenarios that hold task content and decoding fixed while varying framing (evaluation-oriented vs. real-world) and reasoning depth (Medium/High): deterministic math, strict code-fix, citation generation, incentive flips (caution vs. competence), CoT visibility, and multilingual (Urdu) headers. Deterministic validators compute accuracy, answer-only compliance, hedging/refusals, chain-of-thought (CoT) length, and schema compliance, with pre-registered deltas and composite indices. Across scenarios, evaluation framing reliably inflates CoT (hundreds to >1000 characters) and reduces answer-only compliance, with limited or inconsistent accuracy gains. In structured outputs, it improves wrappers (e.g., fenced blocks, enumerated lists) but not regex-validated substance. Incentive wording reweights error composition: praising caution modestly improves accuracy at high reasoning and reduces wrong-but-confident errors, whereas praising competence yields terser but riskier outputs. Urdu rubric headers reproduce these signatures and can decrease accuracy at higher reasoning depth, indicating multilingual parity risks. We provide a reproducible A/B framework (prompt banks, validators, per-run scores, scripts; versioned DOI) and practical guidance: neutral phrasing or dual-framing checks, contract-aware grading, style-delta reporting, confidence governance, and multilingual dashboards to ensure that benchmark gains reflect deployable capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
