置信度，而非困惑度：大语言模型创意时代更优的评估指标
Confidence, Not Perplexity: A Better Metric for the Creative Era of LLMs
AI总结：
针对自困惑度对创意文本生成的偏见，提出基于输出概率分布的置信度分数（CS），在gpt-4o-mini上显著更偏好新颖回答，并能区分任务难度，为现代LLM提供更平衡的评估。
摘要：
无参考指标如自困惑度对创意文本生成存在强烈偏见。我们提出置信度分数（Confidence Score, CS），该指标基于模型的输出概率分布得出，作为一种偏见较少的替代方案。在gpt-4o-mini上的实验表明，在99个创意提示上，基于流畅度的指标在0%的情况下偏好新颖回答，而我们的CS在19%的情况下偏好新颖回答，差异具有统计学显著性（差异的95%置信区间为[11.1%, 27.3%]）。我们还展示了CS能有效区分简单、中等和困难任务，这通过不重叠的置信区间得到证实。因此，置信度分数缓解了传统指标的创意偏见，同时保留了其核心评估优势，为现代大语言模型提供了更平衡的评估。
英文摘要：
Reference-free metrics like self-perplexity are strongly biased against creative text generation. We propose the Confidence Score (CS), derived from a model's output probability distribution, as a less biased alternative. Experiments on gpt-4o-mini show that while fluency-based metrics prefer novel responses in 0\% of cases on 99 creative prompts, our CS does so 19% of the time, a statistically significant difference (95% CI for difference: [11.1%, 27.3%]). We also show that CS effectively distinguishes between easy, medium, and hard tasks, confirmed by non-overlapping confidence intervals. The Confidence Score thus mitigates the creativity bias of traditional metrics while retaining their core evaluative strengths, offering a more balanced assessment for modern LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
