Evaluating the quality of arguments is a crucial aspect of any system leveraging argument mining. However, it is a challenge to obtain reliable and consistent annotations regarding argument quality, as this usually requires domain-specific expertise of the annotators. Even among experts, the assessment of argument quality is often inconsistent due to the inherent subjectivity of this task. In this paper, we study the potential of using state-of-the-art large language models (LLMs) as proxies for argument quality annotators. To assess the capability of LLMs in this regard, we analyze the agreement between model, human expert, and human novice annotators based on an established taxonomy of argument quality dimensions. Our findings highlight that LLMs can produce consistent annotations, with a moderately high agreement with human experts across most of the quality dimensions. Moreover, we show that using LLMs as additional annotators can significantly improve the agreement between annotators. These results suggest that LLMs can serve as a valuable tool for automated argument quality assessment, thus streamlining and accelerating the evaluation of large argument datasets.
Language Model as an Annotator: Unsupervised Context-aware Quality Phrase Generation
College of Economics and Management, Beijing University of Technology（北京工业大学经济与管理学院）；School of Economics and Management, Beihang University（北京航空航天大学经济与管理学院）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）
Estimating Contribution Quality in Online Deliberations Using a Large Language Model
Center on Democracy, Development and the Rule of Law（民主、发展与法治中心）；Stanford University（斯坦福大学）；Management Science and Engineering（管理科学与工程学院）；Cadence Design Systems（楷登电子公司）
Which Argumentative Aspects of Hate Speech in Social Media can be reliably identified?
Universidad de Buenos Aires（布宜诺斯艾利斯大学）；Consejo Nacional de Investigaciones Científicas y Técnicas（阿根廷国家科学技术研究委员会）；Universidad Nacional de Córdoba（科尔多瓦国立大学）；Artificial Intelligence Research Institute (IIIA-CSIC)（人工智能研究所（IIIA-CSIC））；Fundación Via Libre（自由之路基金会）
本文基于 Hateval 语料库和 Periodic Table of Arguments 标注仇恨言论的论证成分，评估各类论证特征的自动识别可靠性，并改进高错误类别。
LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics
Visual Analysis and Perception Lab（视觉分析与感知实验室）；Pioneer Center for AI（先锋人工智能中心）；Center of Excellence for Global Mobility Law（全球移动法律卓越中心）；Department of Computer Science（计算机科学系）
Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm
Institute for Artificial Intelligence Research and Development of Serbia（塞尔维亚人工智能研究与发展研究所）；University of Belgrade（贝尔格莱德大学）；Institute for Philosophy and Social Theory（哲学与社会理论研究所）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau (RPTU)（莱茵兰-普法尔茨州 Kaiserslautern-Landau 应用科技大学）；German Research Centre for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Vilnius Gediminas Technical University（维尔纽斯格迪米纳斯技术大学）；Drexel University（德雷塞尔大学）；University of Montenegro（黑山大学）；Faculty of Dramatic Arts（戏剧艺术学院）；University of Kragujevac（克拉古耶瓦茨大学）；Faculty of Engineering（工程学院）
Uncertainty Decomposition via Cyclical SG-MCMC and Soft-label Learning for Subjective NLP
Faculty of Business and Commerce（商科学部）；Data Science and AI Innovation Research Promotion Center（数据科学与人工智能创新研究促进中心）；Graduate School of Data Science（数据科学研究生院）
Argument Quality Assessment in the Age of Instruction-Following Large Language Models
Leibniz University Hannover（汉诺威莱布尼茨大学）；GESIS（莱布尼茨社会科学研究所）；Heinrich-Heine University Düsseldorf（杜塞尔多夫海因里希·海涅大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；I3S（计算机、信号与系统实验室）；Universität Hamburg（汉堡大学）；University of Richmond（里士满大学）；University of Stuttgart（斯图加特大学）
AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
The University of Hong Kong（香港大学）；Xiamen University（厦门大学）；Microsoft Research Asia（微软亚洲研究院）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Microsoft（微软公司）
Towards a Perspectivist Turn in Argument Quality Assessment
GESIS - Leibniz Institute for the Social Sciences（GESIS-莱布尼茨社会科学研究所）；Leibniz University Hannover（汉诺威莱布尼茨大学）；Heinrich-Heine University Düsseldorf（杜塞尔多夫海因里希·海涅大学）
Identifying Chinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations
Tianjin University（天津大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
目前的观点表达识别（OEI）工作严重依赖于人工构建的训练语料库的质量和规模，这可能是非常难以满足的。众包是解决这一问题的一个可行方案，旨在创建大规模但质量无保证的语料库。在这项工作中，我们调查了中国的OEI与非常嘈杂的众包注释，以非常低的成本构建数据集。根据Zhang et al.（2021），我们通过将所有注释视为人群注释器方面的金标准来训练注释器-适配器模型，并通过使用合成专家（所有注释器的混合物）来测试模型。由于这种用于测试的注释器混合物在训练阶段从未明确建模，因此我们建议通过相关的混合策略生成合成训练样本，以使训练和测试高度一致。在我们构建的数据集上的模拟实验表明，众包是非常有前途的OEI，我们提出的注释者混合可以进一步提高众包建模。
