arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.14501cs.CL
收藏

迈向论证质量评估中的视角主义转向

Towards a Perspectivist Turn in Argument Quality Assessment

GESIS - Leibniz Institute for the Social Sciences(GESIS-莱布尼茨社会科学研究所);Leibniz University Hannover(汉诺威莱布尼茨大学);Heinrich-Heine University Düsseldorf(杜塞尔多夫海因里希·海涅大学)

Julia Romberg, Maximilian Maurer, Henning Wachsmuth, Gabriella Lapesa

AI总结:

本文系统综述论证质量数据集,从标注内容与标注者信息两方面构建多层分类,识别可用于视角主义模型的个体标注数据,并通过试点证明标注者选择的重要性。

摘要:

论证质量的评估依赖于已被充分确立的逻辑、修辞和辩证属性,而这些属性不可避免地具有主观性:可能存在多种有效的评估,并不存在明确的基准真值。这与机器学习领域近来接纳不同视角共存的研究路径相一致。然而,在关于论证质量的NLP研究中,这一潜力在很大程度上仍未得到探索。一个关键原因似乎是合适数据集的可得性尚未得到考察。我们通过对论证质量数据集进行系统综述来填补这一空白。我们将这些数据集归入一个针对两个方面的多层分类体系:(a) 标注了什么:我们收集数据集中涵盖的质量维度,并将其整合为一个总体性分类法,以提高数据集的可比性和互操作性。(b) 谁进行了标注:我们调查了关于标注者给出了哪些信息,从而支持视角主义研究,并为我们关于未来行动的建议奠定基础。为此,我们讨论了适合开发视角主义模型的数据集(即那些包含个体的、未经聚合标注的数据集),并在一项试点研究中展示了受控选择标注者的重要性。

英文摘要:

The assessment of argument quality depends on well-established logical, rhetorical, and dialectical properties that are unavoidably subjective: multiple valid assessments may exist, there is no unequivocal ground truth. This aligns with recent paths in machine learning, which embrace the co-existence of different perspectives. However, this potential remains largely unexplored in NLP research on argument quality. One crucial reason seems to be the yet unexplored availability of suitable datasets. We fill this gap by conducting a systematic review of argument quality datasets. We assign them to a multi-layered categorization targeting two aspects: (a) What has been annotated: we collect the quality dimensions covered in datasets and consolidate them in an overarching taxonomy, increasing dataset comparability and interoperability. (b) Who annotated: we survey what information is given about annotators, enabling perspectivist research and grounding our recommendations for future actions. To this end, we discuss datasets suitable for developing perspectivist models (i.e., those containing individual, non-aggregated annotations), and we showcase the importance of a controlled selection of annotators in a pilot study.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑