SarcasmBench：迈向评估大型语言模型对讽刺的理解
SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding
AI总结：
本文提出SarcasmBench，通过多种提示方法评估LLMs和PLMs在6个讽刺基准上的表现，发现LLMs不及监督PLMs基线，GPT-4表现最佳，且少样本IO提示优于CoT。
摘要：
在大型语言模型（LLMs）时代，诸如情感分析、文本分类等“系统I”任务——即快速、无意识和直觉性的任务——已被认为成功解决。然而，讽刺作为一种微妙的语言现象，常运用夸张和比喻等修辞手法来传达真实情感和意图，其抽象程度高于情感分析。人们日益担忧，在考虑讽刺理解时，关于LLMs成功的论断可能并不完全成立。为解决此问题，我们选取了11个SOTA LLMs和8个SOTA预训练语言模型（PLMs），通过不同提示方法在6个广泛使用的基准数据集上进行了全面评估，即零样本输入/输出（IO）提示、少样本IO提示和思维链（CoT）提示。结果突出了三个关键发现：（1）在6个讽刺基准上，当前LLMs的表现不如基于监督PLMs的讽刺检测基线。这表明仍需付出巨大努力来提升LLMs对人类讽刺的理解。（2）在各种提示方法中，GPT-4始终且显著地优于其他LLMs，平均提升14.0%↑。Claude 3和ChatGPT表现出仅次于GPT-4的性能。（3）少样本IO提示方法优于零样本IO和少样本CoT这两种方法。原因在于，讽刺检测作为一种整体性、直觉性和非理性的认知过程，被认为不遵循逐步逻辑推理，这使得CoT在理解讽刺方面不如其在数学推理任务中有效。
英文摘要：
In the era of large language models (LLMs), the task of ``System I''~-~the fast, unconscious, and intuitive tasks, e.g., sentiment analysis, text classification, etc., have been argued to be successfully solved. However, sarcasm, as a subtle linguistic phenomenon, often employs rhetorical devices like hyperbole and figuration to convey true sentiments and intentions, involving a higher level of abstraction than sentiment analysis. There is growing concern that the argument about LLMs' success may not be fully tenable when considering sarcasm understanding. To address this question, we select eleven SOTA LLMs and eight SOTA pre-trained language models (PLMs) and present comprehensive evaluations on six widely used benchmark datasets through different prompting approaches, i.e., zero-shot input/output (IO) prompting, few-shot IO prompting, chain of thought (CoT) prompting. Our results highlight three key findings: (1) current LLMs underperform supervised PLMs based sarcasm detection baselines across six sarcasm benchmarks. This suggests that significant efforts are still required to improve LLMs' understanding of human sarcasm. (2) GPT-4 consistently and significantly outperforms other LLMs across various prompting methods, with an average improvement of 14.0\%$\uparrow$. Claude 3 and ChatGPT demonstrate the next best performance after GPT-4. (3) Few-shot IO prompting method outperforms the other two methods: zero-shot IO and few-shot CoT. The reason is that sarcasm detection, being a holistic, intuitive, and non-rational cognitive process, is argued not to adhere to step-by-step logical reasoning, making CoT less effective in understanding sarcasm compared to its effectiveness in mathematical reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
