arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.01400cs.CL
收藏

CrowdCounter:一个特定基准类型的多目标反驳言论数据集

CrowdCounter: A benchmark type-specific multi-target counterspeech dataset

Indian Institute of Technology, Kharagpur(印度理工学院卡拉格普尔分校);Indian Institute of Technology, Bhubaneswar(印度理工学院布巴内斯瓦尔分校)

Punyajoy Saha, Abhilash Datta, Abhik Jana, Animesh Mukherjee

AI总结:

本文构建涵盖六类、含3,425对样本的多目标反驳言论数据集CrowdCounter,并评估四种LLM在普通与类型控制提示下的生成效果。

摘要:

反驳言论为在维护言论自由的同时,替代因仇恨言论而封禁或暂停用户提供了一种可行方案。然而,对审核人员或用户而言,撰写有效的反驳言论颇具挑战。因此,当下迫切需要开发用于辅助撰写反驳言论的建议工具。开发此类工具的一个关键挑战在于,现有数据集中的回复缺乏质量与多样性。为此,我们推出了一个新数据集——CrowdCounter,其中包含3,425对仇恨言论—反驳言论,涵盖六种不同的反驳言论类型(共情、幽默、质疑、警告、羞辱、反驳),属同类首创。我们的标注平台在设计上本身就鼓励标注者撰写特定类型、非冗余且高质量的反驳言论。我们在四个大型语言模型上评估了两种生成反驳言论回复的框架:普通提示与类型控制提示。在指标方面,我们从相关性、多样性和质量三个维度评估回复。我们观察到,在普通框架下的不同模型中,Flan-T5表现最佳。特定类型提示能够提升回复的相关性,但可能会降低语言质量。DialoGPT被证明最擅长遵循指令并准确生成特定类型的反驳言论。

英文摘要:

Counterspeech presents a viable alternative to banning or suspending users for hate speech while upholding freedom of expression. However, writing effective counterspeech is challenging for moderators/users. Hence, developing suggestion tools for writing counterspeech is the need of the hour. One critical challenge in developing such a tool is the lack of quality and diversity of the responses in the existing datasets. Hence, we introduce a new dataset - CrowdCounter containing 3,425 hate speech-counterspeech pairs spanning six different counterspeech types (empathy, humor, questioning, warning, shaming, contradiction), which is the first of its kind. The design of our annotation platform itself encourages annotators to write type-specific, non-redundant and high-quality counterspeech. We evaluate two frameworks for generating counterspeech responses - vanilla and type-controlled prompts - across four large language models. In terms of metrics, we evaluate the responses using relevance, diversity and quality. We observe that Flan-T5 is the best model in the vanilla framework across different models. Type-specific prompts enhance the relevance of the responses, although they might reduce the language quality. DialoGPT proves to be the best at following the instructions and generating the type-specific counterspeech accurately.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑