我的会议摘要质量如何？使用多LLM评估器进行质量估计
Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator
AI总结：
针对会议摘要质量自动评估难题，提出基于LLM的MESA框架，通过三步评估与自训练提升与人类判断的相关性，平均比现有方法高0.25，可适应自定义错误指南，适用于少标注数据场景。
摘要：
自然语言生成（NLG）系统生成的会议摘要质量难以自动衡量。ROUGE和BERTScore等现有指标与人类判断的相关性较低，无法捕捉细微错误。近期研究表明，大型语言模型（LLMs）具有更好的上下文理解能力，无需大量人类偏好判断训练即可适应错误定义。然而，当前基于LLM的评估器存在掩盖错误的风险，只能作为弱代理，尽管人类评估成本高且难以跨研究比较，仍是黄金标准。本研究提出MESA，一种基于LLM的框架，采用三步评估：单个错误类型评估、多智能体讨论以完善决策、基于反馈的自训练以优化错误定义理解和与人类判断的对齐。我们证明MESA的组件可实现全面错误检测、一致评分，并能适应自定义错误指南。以GPT-4o为基础，MESA在错误检测中与人类判断的点双列相关系数达中高水平，在反映错误对摘要质量影响的斯皮尔曼和肯德尔相关系数达中等水平，平均比现有方法高0.25。该框架适应自定义错误指南的灵活性使其适用于人类标注数据有限的各类任务。
英文摘要：
The quality of meeting summaries generated by natural language generation (NLG) systems is hard to measure automatically. Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors. Recent studies suggest using large language models (LLMs), which have the benefit of better context understanding and adaption of error definitions without training on a large number of human preference judgments. However, current LLM-based evaluators risk masking errors and can only serve as a weak proxy, leaving human evaluation the gold standard despite being costly and hard to compare across studies. In this work, we present MESA, an LLM-based framework employing a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment. We show that MESA's components enable thorough error detection, consistent rating, and adaptability to custom error guidelines. Using GPT-4o as its backbone, MESA achieves mid to high Point-Biserial correlation with human judgment in error detection and mid Spearman and Kendall correlation in reflecting error impact on summary quality, on average 0.25 higher than previous methods. The framework's flexibility in adapting to custom error guidelines makes it suitable for various tasks with limited human-labeled data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
