评估基于大语言模型的议会辩论总结的论证内容
Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation
AI总结：
本文提出一种评估议会辩论总结的框架，通过计算论证方法评估总结对辩论论证内容的忠实性，以欧洲议会辩论为例验证方法有效性。
摘要：
理解议会中政策辩论和论证的机制是民主过程的基本方面。然而，辩论的规模和复杂性使得外部观众难以参与。同时，大型语言模型（LLMs）已被证明能够实现大规模自动总结。尽管辩论总结可以使议会程序更易接近，但评估这些总结是否忠实传达论证内容仍具挑战性。现有自动总结指标与人类对一致性的判断（即总结与源文本之间的忠实性或对齐程度）相关性差。在本工作中，我们提出了一种评估议会辩论总结的正式框架，该框架将论证结构根植于待辩论的争议提案中。我们的新方法由计算论证驱动，专注于正式属性，即对为支持或反对政策结果所呈现的推理的忠实保存。我们通过欧洲议会辩论的案例研究和相关的LLM驱动总结来展示我们的方法。
英文摘要：
Understanding how policy is debated and justified in parliament is a fundamental aspect of the democratic process. However, the volume and complexity of such debates mean that outside audiences struggle to engage. Meanwhile, Large Language Models (LLMs) have been shown to enable automated summarisation at scale. While summaries of debates can make parliamentary procedures more accessible, evaluating whether these summaries faithfully communicate argumentative content remains challenging. Existing automated summarisation metrics have been shown to correlate poorly with human judgements of consistency (i.e., faithfulness or alignment between summary and source). In this work, we propose a formal framework for evaluating parliamentary debate summaries that grounds argument structures in the contested proposals up for debate. Our novel approach, driven by computational argumentation, focuses the evaluation on formal properties concerning the faithful preservation of the reasoning presented to justify or oppose policy outcomes. We demonstrate our methods using a case-study of debates from the European Parliament and associated LLM-driven summaries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
