Flee the Flaw:通过模板与槽填充标注谬误论证的底层逻辑
Flee the Flaw: Annotating the Underlying Logic of Fallacious Arguments Through Templates and Slot-filling
AI总结:
针对计算论证领域对逻辑错误阐释不足的问题,提出四组可解释非形式逻辑谬误模板,在LOGIC数据集上完成高一致性标注,同时发现SOTA模型谬误模板检测准确率较低,并公开相关资源。
摘要:
计算论证领域的先前研究主要聚焦于论证质量评分,对逻辑错误的阐释关注较少。本研究中,我们针对常见的非形式逻辑谬误引入四组可解释模板,旨在阐明谬误的隐含逻辑。我们基于这些模板,对取自LOGIC数据集的400条谬误论证开展标注研究,取得了较高的一致性得分(Krippendorf's alpha为0.54)与合理的覆盖率(0.83)。最后,我们开展了谬误结构检测实验,发现当前最优(SOTA)语言模型在谬误模板检测上表现不佳(准确率0.47)。为推动谬误相关研究,我们公开了本次的数据集与标注指南。
英文摘要:
Prior research in computational argumentation has mainly focused on scoring the quality of arguments, with less attention on explicating logical errors. In this work, we introduce four sets of explainable templates for common informal logical fallacies designed to explicate a fallacy's implicit logic. Using our templates, we conduct an annotation study on top of 400 fallacious arguments taken from LOGIC dataset and achieve a high agreement score (Krippendorf's alpha of 0.54) and reasonable coverage (0.83). Finally, we conduct an experiment for detecting the structure of fallacies and discover that state-of-the-art language models struggle with detecting fallacy templates (0.47 accuracy). To facilitate research on fallacies, we make our dataset and guidelines publicly available.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。