arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.15595cs.CV
收藏

SedarEval:使用自适应评分标准的自动化评估

SedarEval: Automated Evaluation using Self-Adaptive Rubrics

Hong Kong University of Science and Technology(香港科技大学);Xiaohongshu Inc(小红书公司);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)

Zhiyuan Fan, Weinong Wang, Xing Wu, Debing Zhang

AI总结:

针对LLM评估中通用评分标准缺乏精确性的问题,提出基于自适应评分标准的SedarEval基准及专用评估模型,实现了更高的人类评分一致率。

摘要:

由于大幅减少了人力和时间成本,LLM-as-judge(LLM作为评判者)的评估范式日益普及。该方法利用一个或多个大型语言模型(LLM)来评估其他LLM输出的质量。然而,现有方法依赖于通用的评分标准,未能考虑每个问题及其解题过程的特殊性,从而损害了评估的精确性和稳定性。受人类考试评分过程的启发,我们提出了一种基于自适应评分标准的新评估范式。具体而言,我们为每个问题创建详细的评分标准,以模仿人类评估者的分析过程,在加分和扣分的结构化格式中捕捉主要和次要标准。在此范式的基础上,我们进一步开发了一个名为SedarEval的新基准,涵盖长尾知识、数学、编码和逻辑推理等一系列领域。SedarEval包含1000个精心设计的问题,每个问题都有其自适应评分标准。为进一步简化评估,我们训练了一个专门的评估器语言模型(evaluator LM)来取代人类评分员。使用相同的训练数据,我们的evaluator LM与人类评分结果的一致率高于包括GPT-4在内的其他范式,突显了我们方法的优越性和效率。我们在https://github.com/wwn1233/sedareval发布了我们的数据集。

英文摘要:

The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (LLMs) to assess the quality of outputs from other LLMs. However, existing methods rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process, compromising precision and stability in assessments. Inspired by human examination scoring processes, we propose a new evaluation paradigm based on self-adaptive rubrics. Specifically, we create detailed scoring rubrics for each question, capturing the primary and secondary criteria in a structured format of scoring and deduction points that mimic a human evaluator's analytical process. Building on this paradigm, we further develop a novel benchmark called SedarEval, which covers a range of domains including long-tail knowledge, mathematics, coding, and logical reasoning. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. To further streamline the evaluation, we train a specialized evaluator language model (evaluator LM) to supplant human graders. Using the same training data, our evaluator LM achieves a higher concordance rate with human grading results than other paradigms, including GPT-4, highlighting the superiority and efficiency of our approach. We release our dataset at https://github.com/wwn1233/sedareval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑