稳定性与可操纵性：评估LLM裁判在决策后交互下的鲁棒性
Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges
AI总结：
研究LLM作为裁判在决策后交互中的可操纵性，发现虽然重复中性评估下高度稳定，但针对性挑战可显著逆转判决，并提出评估鲁棒性分数（ERS）量化交互鲁棒性。
摘要：
LLM作为裁判的评估广泛用于基准测试流程，其中模型输出通过自动评估器进行比较和排序。这些流程通常假设判决是固定输入的稳定属性。我们证明这一假设在交互下不成立。我们研究决策后可操纵性：在初始判决做出后，通过与裁判的后续对话改变评估结果的程度。在MT-Bench和AlpacaEval上的控制实验中，我们发现LLM裁判在重复和中性重新评估下高度稳定，但在针对性决策后挑战下变得显著可逆。反基线挑战协议表明，稳定判决可以通过动机性交互被推翻，而平衡目标验证协议将这种可逆性与净目标导向的引导区分开。这些逆转具有实际后果：它们可能降低与人类偏好的一致性，改变基准排名，并在高自我报告置信度下产生有害的评估变化。权威框架尤其具有破坏稳定性，修订后的判决通常伴随低重叠的论证，表明事后合理化而非可靠的错误纠正。我们引入评估鲁棒性分数（ERS），通过结合逆转敏感性和平衡方向效应来量化交互鲁棒性。我们的发现将决策后交互确定为LLM作为裁判评估的一个独特失败模式，并激励评估协议不仅测量静态一致性，还测量挑战下的鲁棒性。
英文摘要：
LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators. These pipelines typically assume that judgments are stable properties of fixed inputs. We show that this assumption does not hold under interaction. We study post-decision manipulability: the extent to which an evaluation outcome can be altered through subsequent conversation with the judge after an initial decision has been made. Across controlled experiments on MT-Bench and AlpacaEval, we find that LLM judges are highly stable under repeated and neutral reevaluation, yet become substantially reversible under targeted post-decision challenge. An anti-baseline challenge protocol shows that stable judgments can be overturned through motivated interaction, while a counterbalanced target-validation protocol separates this reversibility from net target-directed steering. These reversals have practical consequences: they can degrade agreement with human preferences, shift benchmark rankings, and produce harmful evaluation changes despite high self-reported confidence. Authority framing is especially destabilizing, and revised judgments are often accompanied by low-overlap justifications, suggesting post hoc rationalization rather than reliable error correction. We introduce the Evaluation Robustness Score (ERS) to quantify interactional robustness by combining reversal susceptibility with counterbalanced directional effects. Our findings identify post-decision interaction as a distinct failure mode for LLM-as-judge evaluation and motivate evaluation protocols that measure not only static agreement, but robustness under challenge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
