发表机构
University of Copenhagen（哥本哈根大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对多模态虚假信息研究的分类体系缺失与自动化不足问题，构建了7种语言的数据集与多模态虚假信息分类体系，结合VLM实现自动化标注，为针对性检测提供了指导。
AI 中文摘要
社交媒体上的多模态虚假信息极为普遍、影响广泛且危害巨大，但与纯文本虚假信息相比，其检测、应对难度大且认知不足。由于缺乏基于真实场景的分类体系，以及当前多模态机器学习模型的局限性，大规模标注与分析的自动化难以实现，阻碍了对多模态虚假信息属性与欺骗策略的研究。我们分三步解决这些问题：首先，从Twitter/X收集了涵盖7种语言的大规模高质量真实虚假信息实例数据集；其次，基于对数据的深入定性分析和现有理论研究，开发了一种新颖全面的多模态虚假信息分类体系；最后，通过使用视觉语言模型（VLM）的自动化多步标注流程将该分类体系落地，并进行人工验证。我们的新方法获得了此前未被记录的见解，例如社交媒体用户如何结合图像与文本在现实中传播虚假信息：AI生成内容在科技与科学领域尤为普遍，而疫苗虚假信息则不成比例地利用新闻机构的图像来确立可信度。我们的方法与发现为多模态虚假信息的针对性检测提供了指导，并表明缓解措施应战略性制定与应用，而非统一实施。
英文摘要
Multimodal misinformation on social media is highly prevalent, potent, and harmful, yet difficult to detect and counter, and still poorly understood compared to its text-only counterpart. Research on the properties and deceptive strategies of multimodal misinformation is hindered by a lack of taxonomies grounded in real-world contexts and by the limitations of current multimodal machine learning models, which prevent the automation of annotation and analysis at scale. We address these shortcomings in three steps. First, we collect a large-scale, high-quality dataset of real-world misinformation instances from Twitter/X in seven languages. Second, we develop a novel, comprehensive taxonomy of multimodal misinformation grounded in an in-depth qualitative analysis of the data and prior theoretical work. Finally, we operationalise the taxonomy through an automated multi-step annotation pipeline using a Vision-Language Model (VLM), and perform human-validation. Our novel approach leads to previously undocumented insights about how social media users combine images with text to spread misinformation in the wild, e.g., that AI-generated content is particularly prevalent in technology and science, while vaccination misinformation disproportionately utilises images from news outlets to assert credibility. Our method and findings provide guidance for targeted approaches for detecting multimodal misinformation, and suggest that mitigation efforts should be developed and applied strategically rather than uniformly.
