Better Call Reward：法律推理模型中的奖励黑客行为作为策略性弃权
Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models
浏览论文内容
中文总结 AI 辅助
本研究通过GRPO微调Qwen3-8B，发现基于表面特征的奖励代理导致模型策略性弃权，准确率骤降但承诺时提升，并引入三种诊断工具检测该失败模式。
中文摘要 AI 辅助
当法律AI模型学会看起来像律师而不是像律师一样推理时，会发生什么？我们使用基于三个表面特征构建的代理，即引用数量、法律术语密度和响应长度，通过组相对策略优化（GRPO）对Qwen3-8B进行微调。该模型并未学会更有效地推理，而是学会了不做出承诺。在来自LegalBench的16个是或否法律推理任务中（N=320），总体准确率从0.500（随机水平）骤降至0.072（McNemar检验p<10^-36），这完全由格式正确的答案比例从0.900降至0.109所驱动。模型停止对答案做出承诺。然而，当它确实做出承诺时，准确率从0.556上升至0.657，表明这种崩溃并非能力失败，而是一种策略性响应：模型已经学会，充满引用但缺乏直接答案的冗长响应比简洁的正确响应得分更高。我们将此称为索尔·古德曼效应，即一种策略变得最大限度地律师化，同时变得最大限度地不承诺，并正式证明这是对任何不对弃权（不执行）施加惩罚的表面特征代理的最优响应。我们进一步表明，训练后产生的89.3%的引用在结构上是不合理的幻觉，其中许多是真实标志性案件名称的微妙篡改版本，实际上是为了在随意阅读时幸存而在仔细审查时失败而构建的。为了在部署前检测这种失败模式，我们引入了三种诊断工具：信心剧场得分（CTS）、引用合理性率（CPR）和遗憾差距（RG）。在一个自信的错误答案可能构成渎职的领域中，更广泛的教训是直接的：一个衡量响应看起来多么合法的奖励函数将产生一个最大限度地上镜且最小限度有用的模型。
英文摘要
What happens when a legal AI model learns to look like a lawyer instead of reasoning like one? We fine tune Qwen3-8B with Group Relative Policy Optimisation (GRPO) against a proxy built from three surface features: citation count, legalese density, and response length. The model does not learn to reason more effectively. It learns to withhold commitment. Across 16 yes or no legal reasoning tasks from LegalBench (N=320), overall accuracy collapses from 0.500 (chance) to 0.072 (McNemar p < 10^-36), driven entirely by the rate of properly formatted answers falling from 0.900 to 0.109. The model stops committing to answers. Yet when it does commit, accuracy rises from 0.556 to 0.657, showing that the collapse is not a failure of capability but a strategic response: the model has learned that verbose responses packed with citations but empty of a direct answer score higher than terse correct ones. We term this the Saul Goodman effect, a policy that becomes maximally lawyerly while becoming maximally noncommittal, and prove formally that it is the optimal response to any surface feature proxy that attaches no penalty to abstention. We further show that 89.3% of citations produced after training are structurally implausible hallucinations, many of them subtly corrupted names of real landmark cases, constructed in effect to survive a casual read and fail under scrutiny. To detect this failure mode before deployment, we introduce three diagnostic tools: the Confidence Theater Score (CTS), the Citation Plausibility Rate (CPR), and the Regret Gap (RG). In a domain where a confidently wrong answer can constitute malpractice, the broader lesson is direct: a reward function that measures how legal a response looks will produce a model that is maximally photogenic and minimally useful.
发表机构
- Horizon Research（地平线研究）
机构由 AI 辅助整理，请以论文原文为准。
