CriticEval：评估大语言模型作为批评者的能力
CriticEval: Evaluating Large Language Model as Critic
AI总结：
本文提出CriticEval基准，从四个维度、九个任务场景全面可靠地评估LLMs的批评能力，并通过广泛实验验证其可靠性，揭示开源模型潜力及批评能力与任务类型、回复质量等因素的关系。
摘要：
批评能力，即大语言模型（LLMs）识别并纠正回复中缺陷的能力，对于其在自我改进和可扩展监督中的应用至关重要。尽管已有大量研究提出用于评估LLMs批评能力的方法，但其全面性和可靠性仍然有限。为克服这一问题，我们引入了CriticEval，一个旨在全面且可靠地评估LLMs批评能力的新型基准。具体而言，为确保全面性，CriticEval从四个维度、跨九个多样化任务场景评估批评能力。它同时评估标量值和文本形式的批评，针对不同质量的回复。为确保可靠性，大量批评被标注以作为参考，使GPT-4能够可靠地评估文本批评。对开源和闭源LLMs的广泛评估首先验证了CriticEval中评估的可靠性。随后，实验结果表明了开源LLMs的潜力、批评数据集的有效性，以及批评能力与若干关键因素（包括任务类型、回复质量和批评维度）之间若干有趣的关系。
英文摘要：
Critique ability, i.e., the capability of Large Language Models (LLMs) to identify and rectify flaws in responses, is crucial for their applications in self-improvement and scalable oversight. While numerous studies have been proposed to evaluate critique ability of LLMs, their comprehensiveness and reliability are still limited. To overcome this problem, we introduce CriticEval, a novel benchmark designed to comprehensively and reliably evaluate critique ability of LLMs. Specifically, to ensure the comprehensiveness, CriticEval evaluates critique ability from four dimensions across nine diverse task scenarios. It evaluates both scalar-valued and textual critiques, targeting responses of varying quality. To ensure the reliability, a large number of critiques are annotated to serve as references, enabling GPT-4 to evaluate textual critiques reliably. Extensive evaluations of open-source and closed-source LLMs first validate the reliability of evaluation in CriticEval. Then, experimental results demonstrate the promising potential of open-source LLMs, the effectiveness of critique datasets and several intriguing relationships between the critique ability and some critical factors, including task types, response qualities and critique dimensions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
