基于三重间隔目标的后期融合多模态意识形态预测与分析
Late Fusion with Triplet Margin Objective for Multimodal Ideology Prediction and Analysis
摘要：
先前关于意识形态预测的工作主要集中在单一模式上，即，文本或图像。在这项工作中，我们介绍了多模态意识形态预测的任务，其中一个模型预测二进制或五点尺度的意识形态倾向，给定一个文本图像对与政治内容。我们首先收集了五个新的大规模数据集，其中包含英文文档和图像以及它们的意识形态倾向，涵盖了来自美国主流媒体的新闻文章以及Reddit和Twitter的社交媒体帖子。我们对新闻文章进行深入分析，揭示图像内容和使用在政治光谱中的差异。此外，我们进行了广泛的实验和消融研究，证明了针对不同模型组件的预训练目标的有效性。我们表现最好的模型是一种后期融合架构，在多模态内容上使用三重目标进行预训练，其性能比最先进的纯文本模型高出近4%，而没有预训练的强大多模态基线超过3%。
英文摘要：
Prior work on ideology prediction has largely focused on single modalities, i.e., text or images. In this work, we introduce the task of multimodal ideology prediction, where a model predicts binary or five-point scale ideological leanings, given a text-image pair with political content. We first collect five new large-scale datasets with English documents and images along with their ideological leanings, covering news articles from a wide range of US mainstream media and social media posts from Reddit and Twitter. We conduct in-depth analyses of news articles and reveal differences in image content and usage across the political spectrum. Furthermore, we perform extensive experiments and ablation studies, demonstrating the effectiveness of targeted pretraining objectives on different model components. Our best-performing model, a late-fusion architecture pretrained with a triplet objective over multimodal content, outperforms the state-of-the-art text-only model by almost 4% and a strong multimodal baseline with no pretraining by over 3%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
