基于多任务指令的谬误识别算法
Multitask Instruction-based Prompting for Fallacy Recognition
摘要:
谬误被用作看似有效的论点来支持一种立场,并说服听众相信它的有效性。识别谬误对于人类和机器来说都是一项本质上困难的任务。此外,计算模型的一个巨大挑战在于,由于输入格式的差异(例如,问题-答案对,具有谬误片段的句子),体裁(例如,社交媒体、对话、新闻),以及谬误的类型和数量(每个数据集5到18种)。为了解决谬误识别任务,我们将数据集之间的这些差异视为多个任务,并展示了基于T5模型的多任务设置中基于提示的提示如何改善针对特定数据集(如T5,BERT或GPT-3)构建的方法的结果。我们展示了这种多任务提示方法识别跨领域和流派的28个独特谬误的能力,并通过分析每个类(即,谬误类型)的结果。最后,我们分析了注释质量对模型性能的影响,以及用外部知识补充这种方法的可行性。
英文摘要:
Fallacies are used as seemingly valid arguments to support a position and persuade the audience about its validity. Recognizing fallacies is an intrinsically difficult task both for humans and machines. Moreover, a big challenge for computational models lies in the fact that fallacies are formulated differently across the datasets with differences in the input format (e.g., question-answer pair, sentence with fallacy fragment), genre (e.g., social media, dialogue, news), as well as types and number of fallacies (from 5 to 18 types per dataset). To move towards solving the fallacy recognition task, we approach these differences across datasets as multiple tasks and show how instruction-based prompting in a multitask setup based on the T5 model improves the results against approaches built for a specific dataset such as T5, BERT or GPT-3. We show the ability of this multitask prompting approach to recognize 28 unique fallacies across domains and genres and study the effect of model size and prompt choice by analyzing the per-class (i.e., fallacy type) results. Finally, we analyze the effect of annotation quality on model performance, and the feasibility of complementing this approach with external knowledge.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。