MMTF-DES：用于社交媒体数据欲望、情感和情绪分析的多模态Transformer模型融合
MMTF-DES: A Fusion of Multimodal Transformer Models for Desire, Emotion, and Sentiment Analysis of Social Media Data
AI总结：
本文提出统一的多模态Transformer框架MMTF-DES，通过联合微调ViLT和VAuLT模型并采用早期融合策略提取图像-文本对特征，用于识别社交媒体中的人类欲望、情感和情绪。
摘要：
欲望是一组包含语言和认知方面的人类渴望与愿望，它驱动着人类的情感和行为，将人类与其他动物区分开来。理解人类欲望有望成为最迷人和最具挑战性的研究领域之一。它与情感分析和情绪识别任务紧密耦合。它有助于增加人机交互、识别人类情绪智力、理解人际关系以及做出决策。然而，理解人类欲望具有挑战性且研究不足，因为引发欲望的方式在人类之间可能有所不同。由于不同的文化、国家和语言，该任务变得更加困难。先前的研究忽略了图像-文本成对特征表示的使用，而这对于人类欲望理解任务至关重要。在这项研究中，我们提出了一个统一的多模态Transformer框架，采用图像-文本对设置来识别人类欲望、情感和情绪。我们提出方法的核心在于编码器模块，该模块使用两个最先进的多模态Transformer模型构建。这些模型使我们能够提取多样的特征。为了有效地从社交媒体图像和文本对中提取视觉和上下文嵌入特征，我们对两个预训练的多模态Transformer模型进行了联合微调：Vision-and-Language Transformer (ViLT) 和 Vision-and-Augmented-Language Transformer (VAuLT)。随后，我们在这些嵌入特征上使用早期融合策略，以获得图像-文本对的组合多样特征表示。这种整合融合了关于此任务的多样信息，使我们能够从多个视角稳健地感知上下文和图像对。
英文摘要：
Desire is a set of human aspirations and wishes that comprise verbal and cognitive aspects that drive human feelings and behaviors, distinguishing humans from other animals. Understanding human desire has the potential to be one of the most fascinating and challenging research domains. It is tightly coupled with sentiment analysis and emotion recognition tasks. It is beneficial for increasing human-computer interactions, recognizing human emotional intelligence, understanding interpersonal relationships, and making decisions. However, understanding human desire is challenging and under-explored because ways of eliciting desire might be different among humans. The task gets more difficult due to the diverse cultures, countries, and languages. Prior studies overlooked the use of image-text pairwise feature representation, which is crucial for the task of human desire understanding. In this research, we have proposed a unified multimodal transformer-based framework with image-text pair settings to identify human desire, sentiment, and emotion. The core of our proposed method lies in the encoder module, which is built using two state-of-the-art multimodal transformer models. These models allow us to extract diverse features. To effectively extract visual and contextualized embedding features from social media image and text pairs, we conducted joint fine-tuning of two pre-trained multimodal transformer models: Vision-and-Language Transformer (ViLT) and Vision-and-Augmented-Language Transformer (VAuLT). Subsequently, we use an early fusion strategy on these embedding features to obtain combined diverse feature representations of the image-text pair. This consolidation incorporates diverse information about this task, enabling us to robustly perceive the context and image pair from multiple perspectives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
