Check-Eval：一种基于检查表的文本质量评估方法
Check-Eval: A Checklist-based Approach for Evaluating Text Quality
AI总结：
本文提出基于检查表的 Check-Eval 框架，利用 LLM 分两阶段进行无参考或有参考文本质量评估，并在两个基准上验证其优于现有指标。
摘要：
评估大型语言模型（LLM）生成文本的质量仍是一项重大挑战。传统指标往往难以与人类判断良好对齐，尤其是在需要创造性和细微差别的任务中。本文提出 Check-Eval，这是一种利用 LLM 通过基于检查表的方法评估生成文本质量的新型评估框架。Check-Eval 既可作为无参考评估方法，也可作为有参考评估方法，对文本质量提供结构化且可解释的评估。该框架包括两个主要阶段：检查表生成和检查表评估。我们在两个基准数据集上验证了 Check-Eval：Portuguese Legal Semantic Textual Similarity（葡萄牙语法律语义文本相似度）和 SummEval。结果表明，与 G-Eval 和 GPTScore 等现有指标相比，Check-Eval 与人类判断的相关性更高，凸显了其作为自然语言生成任务更可靠、更有效评估框架的潜力。我们的实验代码可在 https://anonymous.4open.science/r/check-eval-0DB4 获取。
英文摘要：
Evaluating the quality of text generated by large language models (LLMs) remains a significant challenge. Traditional metrics often fail to align well with human judgments, particularly in tasks requiring creativity and nuance. In this paper, we propose \textsc{Check-Eval}, a novel evaluation framework leveraging LLMs to assess the quality of generated text through a checklist-based approach. \textsc{Check-Eval} can be employed as both a reference-free and reference-dependent evaluation method, providing a structured and interpretable assessment of text quality. The framework consists of two main stages: checklist generation and checklist evaluation. We validate \textsc{Check-Eval} on two benchmark datasets: Portuguese Legal Semantic Textual Similarity and \textsc{SummEval}. Our results demonstrate that \textsc{Check-Eval} achieves higher correlations with human judgments compared to existing metrics, such as \textsc{G-Eval} and \textsc{GPTScore}, underscoring its potential as a more reliable and effective evaluation framework for natural language generation tasks. The code for our experiments is available at \url{https://anonymous.4open.science/r/check-eval-0DB4}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
