CSEval：利用自动校准大语言模型实现自动化、多维度且无参考的反击言论评估
CSEval: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLMs
AI总结：
本文提出CSEval数据集与Auto-CSEval方法，通过自动校准思维链的大语言模型从四个维度无参考评估反击言论质量，显著优于传统指标。
摘要：
反击言论已成为对抗网络仇恨言论的一种流行且有效的策略，激发了利用语言模型自动化生成反击言论的日益增长的研究兴趣。然而，该领域仍缺乏标准化的评估协议以及与人类判断一致的可靠自动化评估指标。当前的自动评估方法主要基于相似度指标，无法有效捕捉反击言论质量的复杂且相互独立的属性，例如语境相关性、攻击性或论证连贯性。这导致对劳动密集型人工评估的依赖增加，以评估自动反击言论生成方法。为应对这些挑战，我们引入了CSEval，这是一个新颖的数据集和框架，用于从四个维度评估反击言论质量：语境相关性、攻击性、论证连贯性和适宜性。此外，我们提出了用于反击言论评估的自动校准思维链方法（Auto-CSEval），这是一种基于提示的方法，利用大语言模型通过自动校准的思维链（CoT）对反击言论进行评分。我们的实验表明，Auto-CSEval在与人类判断的相关性方面优于ROUGE、METEOR和BertScore等传统指标，表明自动化反击言论评估取得了显著改进。
英文摘要：
Counterspeech has emerged as a popular and effective strategy for combating online hate speech, sparking growing research interest in automating its generation using language models. However, the field still lacks standardised evaluation protocols and reliable automated evaluation metrics that align with human judgement. Current automatic evaluation methods, primarily based on similarity metrics, do not effectively capture the complex and independent attributes of counterspeech quality, such as contextual relevance, aggressiveness, or argumentative coherence. This has led to an increased dependency on labor-intensive human evaluations to assess automated counter-speech generation methods. To address these challenges, we introduce CSEval, a novel dataset and framework for evaluating counterspeech quality across four dimensions: contextual-relevance, aggressiveness, argument-coherence, and suitableness. Furthermore, we propose Auto-Calibrated COT for Counterspeech Evaluation (Auto-CSEval), a prompt-based method with auto-calibrated chain-of-thoughts (CoT) for scoring counterspeech using large language models. Our experiments show that Auto-CSEval outperforms traditional metrics like ROUGE, METEOR, and BertScore in correlating with human judgement, indicating a significant improvement in automated counterspeech evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
