通过裁判之眼：推断的思考轨迹提升LLM评分器的可靠性
Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
AI总结：
通过人与LLM协作框架推断思考轨迹，提升LLM评分器的可靠性及人类与LLM之间的一致性。
摘要：
大型语言模型（LLM）正越来越多地被用作评估任务的评分器。然而，它们的可靠性往往受到限制，尤其是在主观任务中，当人类判断涉及注释标签之外的细微推理时。思考轨迹，即判断背后的原因，信息丰富但难以收集和整理。我们提出了一种人与LLM协作的框架，用于从仅标签的注释中推断出思考轨迹。所提出的框架使用一种简单而有效的拒绝采样方法，在大规模上重建这些轨迹。这些推断出的思考轨迹应用于两个互补的任务：（1）微调开放式的LLM评分器；（2）为专有LLM评分器合成更清晰的注释指南。在多个数据集上，我们的方法显著提高了LLM与人类的一致性。此外，经过细化的注释指南提高了不同LLM模型之间的一致性。这些结果表明，LLM可以作为实际的代理，替代原本无法揭示的人类思考轨迹，使仅标签的数据集扩展为包含思考轨迹增强的资源，从而提高LLM评分器的可靠性。
英文摘要：
Large language models (LLMs) are increasingly used as raters for evaluation tasks. However, their reliability is often limited for subjective tasks, when human judgments involve subtle reasoning beyond annotation labels. Thinking traces, the reasoning behind a judgment, are highly informative but challenging to collect and curate. We present a human-LLM collaborative framework to infer thinking traces from label-only annotations. The proposed framework uses a simple and effective rejection sampling method to reconstruct these traces at scale. These inferred thinking traces are applied to two complementary tasks: (1) fine-tuning open LLM raters; and (2) synthesizing clearer annotation guidelines for proprietary LLM raters. Across multiple datasets, our methods lead to significantly improved LLM-human agreement. Additionally, the refined annotation guidelines increase agreement among different LLM models. These results suggest that LLMs can serve as practical proxies for otherwise unrevealed human thinking traces, enabling label-only corpora to be extended into thinking-trace-augmented resources that enhance the reliability of LLM raters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
