arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.14938cs.CL
收藏

关于LLM的零样本反言论生成

On Zero-Shot Counterspeech Generation by LLMs

Indian Institute of Technology, Kharagpur(印度理工学院克勒格布尔分校);Indian Institute of Technology, Bhubaneswar(印度理工学院布巴内斯瓦尔分校);Universität Hamburg(汉堡大学)

Punyajoy Saha, Aalok Agrawal, Abhik Jana, Chris Biemann, Animesh Mukherjee

AI总结:

本研究首次全面分析了GPT-2、DialoGPT、ChatGPT和FlanT5四种LLM在零样本反言论生成中的表现,并提出三种提示策略以提升生成质量。

摘要:

随着众多大型语言模型(LLM)的出现,此类模型在各种自然语言处理(NLP)应用中的使用正在广泛增加。反言论生成就是这样一个关键任务,人们致力于通过用仇恨言论-反言论对微调LLM来开发生成模型,但这些尝试都没有探索大型语言模型在零样本设置中的内在属性。在这项工作中,我们对四种LLM(即GPT-2、DialoGPT、ChatGPT和FlanT5)在零样本设置下进行反言论生成的性能进行了全面分析,这是同类研究中的首次。对于GPT-2和DialoGPT,我们进一步研究了性能相对于模型大小(小、中、大)的偏差。另一方面,我们提出了三种不同的提示策略来生成不同类型的反言论,并分析了这些策略对模型性能的影响。我们的分析表明,两个数据集的生成质量有所提高(17%),但随着模型大小的增加,毒性也增加了(25%)。考虑模型类型,GPT-2和FlanT5模型在反言论质量方面明显更好,但与DialoGPT相比也具有更高的毒性。ChatGPT在所有指标上生成反言论的能力都比其他模型好得多。在提示方面,我们发现我们提出的策略有助于改善所有模型的反言论生成。

英文摘要:

With the emergence of numerous Large Language Models (LLM), the usage of such models in various Natural Language Processing (NLP) applications is increasing extensively. Counterspeech generation is one such key task where efforts are made to develop generative models by fine-tuning LLMs with hatespeech - counterspeech pairs, but none of these attempts explores the intrinsic properties of large language models in zero-shot settings. In this work, we present a comprehensive analysis of the performances of four LLMs namely GPT-2, DialoGPT, ChatGPT and FlanT5 in zero-shot settings for counterspeech generation, which is the first of its kind. For GPT-2 and DialoGPT, we further investigate the deviation in performance with respect to the sizes (small, medium, large) of the models. On the other hand, we propose three different prompting strategies for generating different types of counterspeech and analyse the impact of such strategies on the performance of the models. Our analysis shows that there is an improvement in generation quality for two datasets (17%), however the toxicity increase (25%) with increase in model size. Considering type of model, GPT-2 and FlanT5 models are significantly better in terms of counterspeech quality but also have high toxicity as compared to DialoGPT. ChatGPT are much better at generating counter speech than other models across all metrics. In terms of prompting, we find that our proposed strategies help in improving counter speech generation across all the models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑