大型语言模型是资深网民吗？一个面向中文网络梗解释的数据集
Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanation
AI总结：
该研究提出中文网络梗解释数据集CHIME，通过梗解释与上下文填空两项任务评估发现，大语言模型对有文化语言差异的梗理解能力不足，性能低于人类水平，旨在推动计算梗理解研究。
摘要：
大型语言模型（LLM）在海量互联网文本上训练，但它们真的理解网络上快速传播的热门内容——也就是通常所说的梗（meme）吗？本文提出了CHIME，即中文网络梗解释数据集（CHinese Internet Meme Explanation）。该数据集收录了中文互联网中流行的短语类梗，并标注了含义、起源、例句、类型等详细信息。为评估LLM是否理解这些梗，我们设计了两项任务。第一项任务评估模型解释给定梗、识别其起源以及生成恰当例句的能力。结果显示，尽管LLM能解释部分梗的含义，但对于带有文化和语言细微差别的梗类型，其性能会显著下降；此外，它们始终难以准确给出梗的起源。第二项任务构建了一组选择题（MCQ），要求LLM选择最合适的梗填入上下文句子的空白处。受评估模型虽能给出正确答案，但性能仍明显低于人类水平。我们已将CHIME公开，希望能推动计算梗理解领域的未来研究。
英文摘要：
Large language models (LLMs) are trained on vast amounts of text from the Internet, but do they truly understand the viral content that rapidly spreads online -- commonly known as memes? In this paper, we introduce CHIME, a dataset for CHinese Internet Meme Explanation. The dataset comprises popular phrase-based memes from the Chinese Internet, annotated with detailed information on their meaning, origin, example sentences, types, etc. To evaluate whether LLMs understand these memes, we designed two tasks. In the first task, we assessed the models' ability to explain a given meme, identify its origin, and generate appropriate example sentences. The results show that while LLMs can explain the meanings of some memes, their performance declines significantly for culturally and linguistically nuanced meme types. Additionally, they consistently struggle to provide accurate origins for the memes. In the second task, we created a set of multiple-choice questions (MCQs) requiring LLMs to select the most appropriate meme to fill in a blank within a contextual sentence. While the evaluated models were able to provide correct answers, their performance remains noticeably below human levels. We have made CHIME public and hope it will facilitate future research on computational meme understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
