arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.09552cs.CL
收藏

大语言模型是少样本训练样本生成器:以谬误识别为例的研究

Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition

Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Columbia University(哥伦比亚大学)

Tariq Alhindi, Smaranda Muresan, Preslav Nakov

AI总结:

针对谬误识别任务中类别不平衡等问题,本研究利用GPT3.5生成合成数据增强稀有类别,探索提示设置、零/少样本场景和补充上下文的作用,实现了跨谬误类型、数据集和生成器的性能提升,相关资源已公开。

摘要:

识别谬误对于保障各领域论证的质量与有效性至关重要。然而,由于数据集中存在的谬误体裁、领域和类型多样,计算式谬误识别面临诸多挑战。这导致了高度多类甚至多标签的设置,且存在严重的类别不平衡问题。本研究旨在通过引入额外上下文、利用大语言模型生成合成数据来增强现有的谬误识别模型,从而提升稀有类别的表征度。我们使用GPT3.5生成合成样本,并检验了提示设置对此的影响。此外,我们探索了零样本和少样本场景,以评估在统一谬误识别框架内,使用生成的样本训练更小模型的有效性。我们还分析了合成数据与现有谬误数据集之间的重叠情况。最后,我们研究了为需要上下文的谬误类型(例如转移话题类谬误)提供补充上下文的作用。我们的评估结果显示,在不同谬误类型、数据集和生成器上,性能均得到了持续提升。代码和合成数据集均已公开。

英文摘要:

Recognizing fallacies is crucial for ensuring the quality and validity of arguments across various domains. However, computational fallacy recognition faces challenges due to the diverse genres, domains, and types of fallacies found in datasets. This leads to a highly multi-class, and even multi-label, setup with substantial class imbalance. In this study, we aim to enhance existing models for fallacy recognition by incorporating additional context and by leveraging large language models to generate synthetic data, thus increasing the representation of the infrequent classes. We experiment with GPT3.5 to generate synthetic examples and we examine the impact of prompt settings for this. Moreover, we explore zero-shot and few-shot scenarios to evaluate the effectiveness of using the generated examples for training smaller models within a unified fallacy recognition framework. Furthermore, we analyze the overlap between the synthetic data and existing fallacy datasets. Finally, we investigate the usefulness of providing supplementary context for detecting fallacy types that need such context, e.g., diversion fallacies. Our evaluation results demonstrate consistent improvements across fallacy types, datasets, and generators. The code and the synthetic datasets are all publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑