评分轮盘：LLM作为裁判框架中的自不一致
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
AI总结：
研究揭示LLM作为裁判框架在不同运行中评分不一致的问题，探讨通过规范指南提升其评估效果的可行性。
摘要：
随着自然语言生成（NLG）的广泛应用，对其正确评估变得困难。最近，使用大型语言模型（LLMs）进行评估获得 traction，因为它们比传统n-gram或嵌入度量更接近人类偏好。在我们的实验中，我们显示LLM裁判在不同运行中分配的评分存在低的内部一致性。这种变异性使评分不一致，在最坏情况下几乎任意，使难以衡量其判断质量。我们量化了不同NLG任务和基准下的不一致性，并检查在遵循适当指南的情况下，是否仍能有效利用LLM裁判。
英文摘要：
As Natural Language Generation (NLG) continues to be widely adopted, properly assessing it has become quite difficult. Lately, using large language models (LLMs) for evaluating these generations has gained traction, as they tend to align more closely with human preferences than conventional n-gram or embedding-based metrics. In our experiments, we show that LLM judges have low intra-rater reliability in their assigned scores across different runs. This variance makes their ratings inconsistent, almost arbitrary in the worst case, making it difficult to measure how good their judgments actually are. We quantify this inconsistency across different NLG tasks and benchmarks and see if judicious use of LLM judges can still be useful following proper guidelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
