SCRum-9：社交媒体谣言的多语言立场分类
SCRum-9: Multilingual Stance Classification over Rumours on Social Media
AI总结：
本文引入覆盖9种语言、含7516条推文的多语言谣言立场分类数据集SCRum-9，通过基准测试模型并探索合成数据应用，发现其可提升小型MLMs性能，还揭示模型预测与人类模糊案例判断的关联，推动相关研究。
摘要：
我们引入SCRum-9，这是最大的多语言谣言分析立场分类数据集，涵盖9种语言，包含来自X平台的7516条推文。SCRum-9超越现有立场分类数据集，覆盖更多语言，将示例与更多事实核查声明（2100条）关联，并包含来自多个标注者的置信度相关标注，以解释标注者内部和之间的变异性。每种语言至少由两名母语者进行标注，总标注时长超过405小时，补偿金额达8150美元。此外，SCRum-9用于在上下文学习（ICL）和微调设置中对5个大型语言模型（LLMs）和2个多语言掩码语言模型（MLMs）进行基准测试。本文还通过探索多语言合成数据在谣言立场分类中的应用进行创新，结果显示即使ICL性能较弱的LLMs也能生成有价值的合成数据用于微调小型MLMs，使其性能超过LLMs的零样本ICL。最后，我们研究了模型预测与人类对模糊案例不确定性的关系，发现模型预测通常匹配标注者分配的第二选择标签，而非完全偏离人类判断。SCRum-9已向研究社区公开，有望推动社交媒体误导性叙事多语言分析的进一步研究。
英文摘要：
We introduce SCRum-9, the largest multilingual Stance Classification dataset for Rumour analysis in 9 languages, containing 7,516 tweets from X. SCRum-9 goes beyond existing stance classification datasets by covering more languages, linking examples to more fact-checked claims (2.1k), and including confidence-related annotations from multiple annotators to account for intra- and inter-annotator variability. Annotations were made by at least two native speakers per language, totalling more than 405 hours of annotation and 8,150 dollars in compensation. Further, SCRum-9 is used to benchmark five large language models (LLMs) and two multilingual masked language models (MLMs) in In-Context Learning (ICL) and fine-tuning setups. This paper also innovates by exploring the use of multilingual synthetic data for rumour stance classification, showing that even LLMs with weak ICL performance can produce valuable synthetic data for fine-tuning small MLMs, enabling them to achieve higher performance than zero-shot ICL in LLMs. Finally, we examine the relationship between model predictions and human uncertainty on ambiguous cases finding that model predictions often match the second-choice labels assigned by annotators, rather than diverging entirely from human judgments. SCRum-9 is publicly released to the research community with potential to foster further research on multilingual analysis of misleading narratives on social media.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
