HD-Eval：通过分层标准分解对齐大语言模型评估器
HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition
AI总结：
HD-Eval提出通过分层标准分解迭代对齐LLM评估器与人类偏好，以白盒聚合器实现高效可解释的细粒度评估，并在多个领域验证其优越性。
摘要：
大语言模型（LLMs）已成为昂贵人工评估的一种有前景的替代方案。然而，基于LLM的评估的对齐性和覆盖范围往往受到评估提示与标准的范围及潜在偏差的限制。为应对这一挑战，我们提出HD-Eval，一个通过分层标准分解（Hierarchical Criteria Decomposition）迭代地将基于LLM的评估器与人类偏好对齐的新框架。HD-Eval继承了人类专家评估思维的精髓，通过将给定评估任务分解为更细粒度的标准、根据估计的人类偏好进行聚合、利用归因剪除不重要的标准，并进一步分解重要标准，从而增强基于LLM的评估器的对齐性。通过在迭代对齐训练过程中整合这些步骤，我们获得了标准的分层分解，能够在多个粒度层级上全面捕捉自然语言的各个方面。HD-Eval以白盒方式实现，其由人类偏好引导的聚合器训练高效，且比仅依赖提示更可解释，同时其独立于模型参数的特性使其适用于闭源LLMs。在三个评估领域上的大量实验表明，HD-Eval在进一步对齐最先进的评估器方面具有优越性，并为评估结果及任务本身的解释提供了更深入的洞见。
英文摘要：
Large language models (LLMs) have emerged as a promising alternative to expensive human evaluations. However, the alignment and coverage of LLM-based evaluations are often limited by the scope and potential bias of the evaluation prompts and criteria. To address this challenge, we propose HD-Eval, a novel framework that iteratively aligns LLM-based evaluators with human preference via Hierarchical Criteria Decomposition. HD-Eval inherits the essence from the evaluation mindset of human experts and enhances the alignment of LLM-based evaluators by decomposing a given evaluation task into finer-grained criteria, aggregating them according to estimated human preferences, pruning insignificant criteria with attribution, and further decomposing significant criteria. By integrating these steps within an iterative alignment training process, we obtain a hierarchical decomposition of criteria that comprehensively captures aspects of natural language at multiple levels of granularity. Implemented as a white box, the human preference-guided aggregator is efficient to train and more explainable than relying solely on prompting, and its independence from model parameters makes it applicable to closed-source LLMs. Extensive experiments on three evaluation domains demonstrate the superiority of HD-Eval in further aligning state-of-the-art evaluators and providing deeper insights into the explanation of evaluation results and the task itself.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
