基于微调自注意和迁移学习的Web信息多模态融合准确性分析
Multi-modal Fusion using Fine-tuned Self-attention and Transfer Learning for Veracity Analysis of Web Information
摘要:
自在线社交网络出现以来,错误信息和假新闻的危害已经升级了许多倍。人的意识和决策能力受到操纵、捏造、有偏见或未经证实的新闻帖子的负面影响。因此,设计准确性分析系统以检测多种数据模态中的虚假信息内容具有很高的要求。在试图找到一个复杂的解决方案,这个关键问题,我们提出了一个架构,同时考虑文本和视觉属性的数据。在完成数据预处理之后,使用单独的深度学习模型从训练数据中提取文本和图像特征。使用BERT和ALBERT语言模型从文本中进行特征提取,这些语言模型利用了使用深度自我注意力机制的双向训练的好处。Inception-ResNet-v2深度神经网络模型用于图像数据以执行任务。提出的框架重点关注两个独立的多模态融合架构BERT和Inception-ResNet-v2以及ALBERT和Inception-ResNet-v2。文本和视觉分支的多模态融合进行了广泛的实验和分析,使用级联的特征向量和加权平均的概率命名为早期融合和后期融合分别。三个公开的广泛接受的数据集All Data,Weibo和MediaEval 2016,分别包含英语新闻文章,中文新闻文章和Tweets,以便我们设计的框架的结果可以正确测试并与该领域以前的显着工作进行比较。
英文摘要:
The nuisance of misinformation and fake news has escalated many folds since the advent of online social networks. Human consciousness and decision-making capabilities are negatively influenced by manipulated, fabricated, biased or unverified news posts. Therefore, there is a high demand for designing veracity analysis systems to detect fake information contents in multiple data modalities. In an attempt to find a sophisticated solution to this critical issue, we proposed an architecture to consider both the textual and visual attributes of the data. After the data pre-processing is done, text and image features are extracted from the training data using separate deep learning models. Feature extraction from text is done using BERT and ALBERT language models that leverage the benefits of bidirectional training of transformers using a deep self-attention mechanism. The Inception-ResNet-v2 deep neural network model is employed for image data to perform the task. The proposed framework focused on two independent multi-modal fusion architectures of BERT and Inception-ResNet-v2 as well as ALBERT and Inception-ResNet-v2. Multi-modal fusion of textual and visual branches is extensively experimented and analysed using concatenation of feature vectors and weighted averaging of probabilities named as Early Fusion and Late Fusion respectively. Three publicly available broadly accepted datasets All Data, Weibo and MediaEval 2016 that incorporates English news articles, Chinese news articles, and Tweets correspondingly are used so that our designed framework's outcomes can be properly tested and compared with previous notable work in the domain.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。