ConQRet：使用LLM评判者对检索增强论证进行细粒度评估的基准测试
ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges
AI总结：
针对检索增强论证缺乏整体细粒度评估的问题，本文提出多LLM评判者自动评估方法，并构建基于真实网站证据的ConQRet基准以衡量检索、论证质量与依据性。
摘要：
计算论证涉及为堕胎禁令和疫苗接种等争议性话题生成答案或摘要，在当今两极分化的环境中已变得日益重要。先进的LLM能力有潜力通过检索增强论证（Retrieval-Augmented Argumentation，RAArg）为此类问题提供细致、基于证据的答案，利用现实世界证据形成高质量、有依据的论证。然而，评估RAArg仍然具有挑战性，因为对于复杂话题上冗长而复杂的答案，人工评估成本高昂且实施困难。同时，复用现有论证数据集已不再足够，因为这些数据集缺少冗长、复杂的论证以及来自可能具有误导性来源的现实证据，从而限制了对检索有效性和论证质量的整体评估。为弥补这些空白，我们研究了使用多个细粒度LLM评判者的自动化评估方法，相比传统的单一分数指标，甚至此前报告的人工众包，提供更好且更具可解释性的评估。为验证所提出的技术，我们引入了ConQRet这一新基准，其特点是围绕受争议话题构建、由人工撰写的冗长复杂论证，并以现实世界网站为依据，从而能够跨检索有效性、论证质量和依据性进行详尽评估。我们在一个先前数据集和新的ConQRet基准上验证了我们的LLM评判者。我们提出的LLM评判者和ConQRet基准能够推动计算论证的快速发展，并可自然扩展到其他复杂的检索增强生成任务。
英文摘要：
Computational argumentation, which involves generating answers or summaries for controversial topics like abortion bans and vaccination, has become increasingly important in today's polarized environment. Sophisticated LLM capabilities offer the potential to provide nuanced, evidence-based answers to such questions through Retrieval-Augmented Argumentation (RAArg), leveraging real-world evidence for high-quality, grounded arguments. However, evaluating RAArg remains challenging, as human evaluation is costly and difficult for complex, lengthy answers on complicated topics. At the same time, re-using existing argumentation datasets is no longer sufficient, as they lack long, complex arguments and realistic evidence from potentially misleading sources, limiting holistic evaluation of retrieval effectiveness and argument quality. To address these gaps, we investigate automated evaluation methods using multiple fine-grained LLM judges, providing better and more interpretable assessments than traditional single-score metrics and even previously reported human crowdsourcing. To validate the proposed techniques, we introduce ConQRet, a new benchmark featuring long and complex human-authored arguments on debated topics, grounded in real-world websites, allowing an exhaustive evaluation across retrieval effectiveness, argument quality, and groundedness. We validate our LLM Judges on a prior dataset and the new ConQRet benchmark. Our proposed LLM Judges and the ConQRet benchmark can enable rapid progress in computational argumentation and can be naturally extended to other complex retrieval-augmented generation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
