基于似然的大语言模型评估偏差缓解方法
Likelihood-based Mitigation of Evaluation Bias in Large Language Models
AI总结：
本文研究大语言模型评估中的似然偏差问题，提出利用高偏差实例作为少样本示例进行上下文学习来缓解偏差，实验证明该方法能有效减少偏差并显著提升评估性能。
摘要：
大语言模型（LLMs）被广泛用作自动化指标来评估自然语言生成任务。然而，似然作为衡量大语言模型对句子合理性判断的指标，可能会因词序和句子结构等表面差异而发生变化。因此，如果使用大语言模型进行评估，可能存在似然偏差：它们可能高估似然较高的句子，而低估似然较低的句子。本文研究了基于大语言模型的评估器中似然偏差的存在及其影响，并提出了一种缓解似然偏差的方法。我们的方法利用高偏差实例作为上下文学习的少样本示例。我们在数据到文本和语法错误纠正任务的评估实验表明，所测试的多个大语言模型均表现出似然偏差。此外，我们提出的方法成功缓解了这种偏差，并显著提升了评估性能（以模型与人类评分的相关性衡量）。
英文摘要：
Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics. However, the likelihood, a measure of LLM's plausibility for a sentence, can vary due to superficial differences in sentences, such as word order and sentence structure. It is therefore possible that there might be a likelihood bias if LLMs are used for evaluation: they might overrate sentences with higher likelihoods while underrating those with lower likelihoods. In this paper, we investigate the presence and impact of likelihood bias in LLM-based evaluators. We also propose a method to mitigate the likelihood bias. Our method utilizes highly biased instances as few-shot examples for in-context learning. Our experiments in evaluating the data-to-text and grammatical error correction tasks reveal that several LLMs we test display a likelihood bias. Furthermore, our proposed method successfully mitigates this bias, also improving evaluation performance (in terms of correlation of models with human scores) significantly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
