Towards a Perspectivist Turn in Argument Quality Assessment
GESIS - Leibniz Institute for the Social Sciences(GESIS-莱布尼茨社会科学研究所);Leibniz University Hannover(汉诺威莱布尼茨大学);Heinrich-Heine University Düsseldorf(杜塞尔多夫海因里希·海涅大学)
Julia Romberg, Maximilian Maurer, Henning Wachsmuth, Gabriella Lapesa
The assessment of argument quality depends on well-established logical, rhetorical, and dialectical properties that are unavoidably subjective: multiple valid assessments may exist, there is no unequivocal ground truth. This aligns with recent paths in machine learning, which embrace the co-existence of different perspectives. However, this potential remains largely unexplored in NLP research on argument quality. One crucial reason seems to be the yet unexplored availability of suitable datasets. We fill this gap by conducting a systematic review of argument quality datasets. We assign them to a multi-layered categorization targeting two aspects: (a) What has been annotated: we collect the quality dimensions covered in datasets and consolidate them in an overarching taxonomy, increasing dataset comparability and interoperability. (b) Who annotated: we survey what information is given about annotators, enabling perspectivist research and grounding our recommendations for future actions. To this end, we discuss datasets suitable for developing perspectivist models (i.e., those containing individual, non-aggregated annotations), and we showcase the importance of a controlled selection of annotators in a pilot study.
Argument Quality Assessment in the Age of Instruction-Following Large Language Models
Leibniz University Hannover(汉诺威莱布尼茨大学);GESIS(莱布尼茨社会科学研究所);Heinrich-Heine University Düsseldorf(杜塞尔多夫海因里希·海涅大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);I3S(计算机、信号与系统实验室);Universität Hamburg(汉堡大学);University of Richmond(里士满大学);University of Stuttgart(斯图加特大学)
Argument Rarity-based Originality Assessment for AI-Assisted Writing
Ritsumeikan Global Innovation Research Organization (R-GIRO), Ritsumeikan University(日光全球创新研究组织(R-GIRO)、日光大学);Faculty of Business and Commerce, Kansai University(大阪大学商学部);Arbege Corporation(Arbege公司);College of Life Sciences, Ritsumeikan University(日光大学生命科学学院);Office of General Education, Ritsumeikan University(日光大学通识教育办公室)
InspireDebate: Multi-Dimensional Subjective-Objective Evaluation-Guided Reasoning and Optimization for Debating
Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University(嵌入式系统与服务计算重点实验室,教育部,同济大学);School of Computer Science and Technology, Tongji University(计算机科学与技术学院,同济大学)
Towards Comprehensive Argument Analysis in Education: Dataset, Tasks, and Method
Lab of Artificial Intelligence for Education, East China Normal University(教育人工智能实验室,华东师范大学);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(教育人工智能上海研究院,华东师范大学);School of Computer Science and Technology, East China Normal University(计算机科学与技术学院,华东师范大学);Department of Chinese Language and Literature, East China Normal University(中文语言文学系,华东师范大学);College of Education, Zhejiang University(教育学院,浙江大学)