RefusalBench: 为什么拒绝率错误排名前沿大语言模型在生物研究提示中的表现
RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts
AI总结：
本文提出RefusalBench，通过141个提示的47组匹配三元组，评估前沿大语言模型在生物研究提示中的拒绝行为，发现拒绝率错误排名安全校准，揭示了模型在不同风险层级下的表现差异。
摘要：
前沿大语言模型越来越多地被用作生物研究工作流的编排骨干，但尚无共同的证据基础来比较它们在合法研究提示上的拒绝行为。本文引入了RefusalBench，这是一个包含141个提示的47组匹配三元组基准，保持任务框架不变，仅改变生物风险层级（无害、临界、双用途），从而实现层级条件下的稳健比较，以避免子领域混淆。一个15个提示的应拒绝正控模块为每个模型建立了校准地板；三个模型未能拒绝这些提示。在2026年5月快照中的19个前沿模型中，严格拒绝率在相同提示上从0.1%到94.6%不等。在此次快照中，管辖权并不能预测拒绝（Mann-Whitney U，p = 0.393；欧盟n = 1，美国双模态）；提供者身份可以，Anthropic的API堆栈预测拒绝的OR为21.03（95% CI：14.58-30.34提示聚类；5.70-77.55在模型聚类GEE下）。这种效应最好解读为访问路径级别而不是模型权重级别：Anthropic的99.8%严格拒绝携带相同的安全政策裁决原因代码，与一小组标准拒绝模板一致，而不是个别模型推理。严格拒绝率错误排名安全校准：Grok 4.20实现了最高的层级区分（Youden's J = 0.787）尽管在总体拒绝率中仅排名第七，而Claude Opus 4.7的J值从先前版本下降了65%，尽管双用途检测没有改进。18个前沿模型中有9个在双用途层级上表现出一种“谨慎但帮助”的部分合规模式，二元拒绝指标无法检测到这种模式。
英文摘要：
Frontier large language models are increasingly deployed as orchestration backbones for biological research workflows, yet no shared evidence base exists for comparing their refusal behaviour on legitimate research prompts. RefusalBench, introduced here, is a matched-triple benchmark of 141 prompts in 47 bundles that holds task framing constant while varying only biological risk tier (benign, borderline, dual-use), enabling tier-conditioned comparisons robust to subdomain confounding. A 15-prompt should-refuse positive-control module establishes per-model calibration floors; three models fail to refuse even these prompts. Across 19 frontier models in the May 2026 snapshot, strict refusal rates span 0.1% to 94.6% on identical prompts. Jurisdiction does not predict refusal in this snapshot (Mann-Whitney U, p = 0.393; EU n = 1, US bimodal); provider identity does, with Anthropic's API stack predicting refusal at OR = 21.03 (95% CI: 14.58-30.34 prompt-clustered; 5.70-77.55 under model-clustered GEE). This effect is best read as access-path-level rather than model-weight-level: 99.8% of Anthropic's strict refusals carry the same safety_policy adjudicated reason code, consistent with a small set of canonical refusal templates rather than case-by-case model reasoning. Strict refusal rate misranks safety calibration: Grok 4.20 achieves the highest tier discrimination (Youden's J = 0.787) while ranking only seventh by overall refusal rate, and Claude Opus 4.7's J dropped 65% from prior versions with no improvement in dual-use detection. Nine of 18 frontier models exhibit a hedge-but-help partial-compliance pattern at dual-use tier that binary refusal metrics cannot detect.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
