校准评估器：概率校准能否缓解LLM智能体反馈回路中的偏好耦合？
Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?
AI总结：
研究通过概率校准评估器的成对判断来缓解LLM智能体反馈回路中的偏好耦合，实验表明校准将耦合系数降低20-49%，JS散度降低45-67%。
摘要：
当大型语言模型（LLM）智能体通过评估器反馈调整其行为时，系统性的评估器偏差会传播到智能体学习到的策略分布中——这一现象被称为评估器偏好耦合。先前的工作记录了这种耦合并建立了诊断框架（EPC）来测量它，但尚未研究校准技术是否能缓解该效应。我们首次研究了评估器校准作为缓解手段：将概率校准应用于评估器的成对判断，以减少虚假偏好传播。在一项受控的受试者内实验（N=5）中，使用DeepSeek-V4-Pro作为执行器、GLM5.2作为评估器，比较了标准二元TTRL（胜/负）与置信度校准TTRL（概率加权更新），我们发现校准将耦合系数γ降低了20-49%，将Jensen-Shannon散度降低了45-67%。对称LR对照实验证实该效应并非由于更新不对称性降低所致。我们发布了校准后的TTRL协议，并推荐将其作为LLM-as-judge部署流水线中的轻量级缓解措施。
英文摘要：
When large language model (LLM) agents adapt their behavior through evaluator feedback, systematic evaluator biases propagate into the agent's learned strategy distribution - a phenomenon termed evaluator preference coupling. Prior work has documented this coupling and established a diagnostic framework (EPC) to measure it, but has not investigated whether calibration techniques can mitigate the effect. We present the first study of evaluator calibration as mitigation: applying probability calibration to the evaluator's pairwise judgments to reduce spurious preference propagation. In a controlled within-subjects experiment (N=5) comparing standard binary TTRL (win/loss) with confidence-calibrated TTRL (probability-weighted updates) using DeepSeek-V4-Pro as executor and GLM5.2 as evaluator, we find that calibration reduces the coupling coefficient gamma by 20-49% and Jensen-Shannon divergence by 45-67%. A symmetric-LR control confirms the effect is not due to reduced update asymmetry. We release the calibrated TTRL protocol and recommend it as a lightweight mitigation for LLM-as-judge deployment pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
