LLM作为评判者中设置温度的必要性
The Necessity of Setting Temperature in LLM-as-a-Judge
AI总结：
系统研究温度对LLM评判行为的影响，发现高温降低一致性但暴露不确定性，低温适合稳定任务，高温适合复杂场景，建议温度作为任务相关的设计选择。
摘要：
使用大型语言模型（LLM）作为评判者来评估模型输出已成为自动化评估的重要范式。然而，在LLM作为评判者的设置中，解码温度的选择在很大程度上仍然是经验性的，缺乏关于其影响的系统证据。为了解决这一差距，我们系统研究了温度如何影响不同LLM评判模型、提示策略和评估范式下的评判行为。我们的结果表明，较高的温度通常会降低评判一致性并增加格式错误，同时也会暴露潜在的不确定性，这种不确定性在低温解码下往往被抑制，尤其是在模糊案例中。进一步的分析表明，较高的温度可以作为探索机制，并可能提高复杂或不确定评估场景中的评判性能。总体而言，低温设置更适合优先考虑稳定性和可重复性的任务，而高温设置更适合涉及大量模糊性或复杂性的场景，在这些场景中，探索评判者的决策空间是有益的。这些发现表明，在LLM作为评判者的系统中，温度不应被视为固定的超参数，而应被视为可控的、任务相关的设计选择，它调节了可靠性与探索之间的权衡。
英文摘要：
Using large language models (LLMs) as judges for evaluating model outputs has emerged as an important paradigm for automated evaluation. However, the choice of decoding temperature in LLM-as-a-judge settings is still largely chosen empirically, with limited systematic evidence on its impact. To address this gap, we conduct a systematic study of how temperature affects judgment behavior across different LLM judge models, prompting strategies, and evaluation paradigms. Our results show that higher temperatures generally decrease judgment consistency and increase formatting errors, while also exposing latent uncertainty that tends to remain suppressed under low-temperature decoding, particularly in ambiguous cases. Further analysis suggests that higher temperatures can serve as an exploratory mechanism and may improve judging performance in complex or uncertain evaluation scenarios. Overall, low-temperature settings are better suited to tasks that prioritize stability and reproducibility, whereas higher-temperature settings are more appropriate for scenarios involving substantial ambiguity or complexity, where exploration of the judge's decision space is beneficial. These findings suggest that, in LLM-as-a-judge systems, temperature should be treated not as a fixed hyperparameter, but as a controllable, task-dependent design choice that mediates the trade-off between reliability and exploration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
