你站在哪一边？用于端到端论证摘要与评估的多任务数据集
Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation
AI总结：
本研究提出一个多任务论证挖掘数据集，涵盖主张识别、证据排序、摘要生成及质量评估，发现LLMs在端到端任务链上性能显著下降，为未来研究提供挑战与基准。
摘要：
随着大型语言模型（LLMs）的最新进展，构建一个帮助人们综合有说服力论点的自动化辩论系统已不再不可行。以往的工作尝试通过整合多个组件来完成这一任务。在我们的工作中，我们引入了一个论证挖掘数据集，该数据集捕捉了为辩论准备议论文的端到端过程，涵盖了以下任务：主张和证据识别（任务1 ED）、证据说服力排序（任务2 ECR）、议论文摘要和人类偏好排序（任务3 ASR）以及基于人类对论证质量维度的反馈，对生成的文章进行自动化评估的度量学习（任务4 SQE）。我们的数据集包含14k个主张示例，这些主张完全标注了支持上述任务的各种属性。我们评估了针对每个任务的多个生成基线，包括具有代表性的LLMs。我们发现，虽然这些模型在我们的基准测试中对个别任务显示出有前景的结果，但它们在依次执行所有四个任务的端到端性能上显著下降，无论是在自动化度量还是在以人为中心的评估中。我们提出的数据集所带来的这一挑战，激励了未来对端到端论证挖掘和摘要的研究。该项目的代码库可在https://github.com/HaoBytes/ArgSum-Datatset获取。
英文摘要：
With the recent advances of large language models (LLMs), it is no longer infeasible to build an automated debate system that helps people to synthesise persuasive arguments. Previous work attempted this task by integrating multiple components. In our work, we introduce an argument mining dataset that captures the end-to-end process of preparing an argumentative essay for a debate, which covers the tasks of claim and evidence identification (Task 1 ED), evidence convincingness ranking (Task 2 ECR), argumentative essay summarisation and human preference ranking (Task 3 ASR) and metric learning for automated evaluation of resulting essays, based on human feedback along argument quality dimensions (Task 4 SQE). Our dataset contains 14k examples of claims that are fully annotated with the various properties supporting the aforementioned tasks. We evaluate multiple generative baselines for each of these tasks, including representative LLMs. We find, that while they show promising results on individual tasks in our benchmark, their end-to-end performance on all four tasks in succession deteriorates significantly, both in automated measures as well as in human-centred evaluation. This challenge presented by our proposed dataset motivates future research on end-to-end argument mining and summarisation. The repository of this project is available at https://github.com/HaoBytes/ArgSum-Datatset
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
