RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification
School of Computer Science, Fudan University（复旦大学计算机科学学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；nyonic；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言学研究院）
Junjie Ye, Jie Zhou, Junfeng Tian, Rui Wang, Qi Zhang, Tao Gui, Xuanjing Huang
Recently, Target-oriented Multimodal Sentiment Classification (TMSC) has gained significant attention among scholars. However, current multimodal models have reached a performance bottleneck. To investigate the causes of this problem, we perform extensive empirical evaluation and in-depth analysis of the datasets to answer the following questions: Q1: Are the modalities equally important for TMSC? Q2: Which multimodal fusion modules are more effective? Q3: Do existing datasets adequately support the research? Our experiments and analyses reveal that the current TMSC systems primarily rely on the textual modality, as most of targets' sentiments can be determined solely by text. Consequently, we point out several directions to work on for the TMSC task in terms of model design and dataset construction. The code and data can be found in https://github.com/Junjie-Ye/RethinkingTMSC.
Tsinghua University（清华大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
This paper focuses on learning domain-oriented language models driven by end tasks, which aims to combine the worlds of both general-purpose language models (such as ELMo and BERT) and domain-specific language understanding. We propose DomBERT, an extension of BERT to learn from both in-domain corpus and relevant domain corpora. This helps in learning domain language models with low-resources. Experiments are conducted on an assortment of tasks in aspect-based sentiment analysis, demonstrating promising results.
Aspect-level sentiment classification (ALSC) and aspect oriented opinion words extraction (AOWE) are two highly relevant aspect-based sentiment analysis (ABSA) subtasks. They respectively aim to detect the sentiment polarity and extract the corresponding opinion words toward a given aspect in a sentence. Previous works separate them and focus on one of them by training neural models on small-scale labeled data, while neglecting the connections between them. In this paper, we propose a novel joint model, Opinion Transmission Network (OTN), to exploit the potential bridge between ALSC and AOWE t…
The Chinese University of Hong Kong（香港中文大学）；Tencent AI Lab（腾讯人工智能实验室）
Target-oriented sentiment classification aims at classifying sentiment polarities over individual opinion targets in a sentence. RNN with attention seems a good fit for the characteristics of this task, and indeed it achieves the state-of-the-art performance. After re-examining the drawbacks of attention mechanism and the obstacles that block CNN to perform well in this classification task, we propose a new model to overcome these issues. Instead of attention, our model employs a CNN layer to extract salient features from the transformed word representations originated from a bi-directional RN…
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Yahoo! Research（雅虎研究院）；Huawei Research America（华为美国研究院）
While existing machine learning models have achieved great success for sentiment classification, they typically do not explicitly capture sentiment-oriented word interaction, which can lead to poor results for fine-grained analysis at the snippet level (a phrase or sentence). Factorization Machine provides a possible approach to learning element-wise interaction for recommender systems, but they are not directly applicable to our task due to the inability to model contexts and word sequences. In this work, we develop two Position-aware Factorization Machines which consider word interaction, co…
University of Wolverhampton（伍尔弗汉普顿大学）；University of Surrey（萨里大学）
Social media companies as well as authorities make extensive use of artificial intelligence (AI) tools to monitor postings of hate speech, celebrations of violence or profanity. Since AI software requires massive volumes of data to train computers, Machine Translation (MT) of the online content is commonly used to process posts written in several languages and hence augment the data needed for training. However, MT mistakes are a regular occurrence when translating sentiment-oriented user-generated content (UGC), especially when a low-resource language is involved. The adequacy of the whole pr…
Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；University of Trento（特伦托大学）
We propose a neural machine translation (NMT) approach that, instead of pursuing adequacy and fluency ("human-oriented" quality criteria), aims to generate translations that are best suited as input to a natural language processing component designed for a specific downstream task (a "machine-oriented" criterion). Towards this objective, we present a reinforcement learning technique based on a new candidate sampling strategy, which exploits the results obtained on the downstream task as weak feedback. Experiments in sentiment classification of Twitter data in German and Italian show that feedi…
IBM Corporation（国际商业机器公司）；North Carolina State University（北卡罗来纳州立大学）
Opinionated text often involves attributes such as authorship and location that influence the sentiments expressed for different aspects. We posit that structural and semantic correspondence is both prevalent in opinionated text, especially when associated with attributes, and crucial in accurately revealing its latent aspect and sentiment structure. However, it is not recognized by existing approaches. We propose Trait, an unsupervised probabilistic model that discovers aspects and sentiments from text and associates them with different attributes. To this end, Trait infers and leverages stru…
Target-oriented Multimodal Sentiment Classification with Counterfactual-enhanced Debiasing
School of Computer, Electronics and Information（计算机、电子与信息学院）；Guangxi University（广西大学）；Guangxi Key Laboratory of Multimedia Communications（广西多媒体通信与网络技术重点实验室）；School of Sociology and Anthropology（社会学与人类学学院）；Sun Yat-sen University（中山大学）
Target-oriented sentiment classification is a fine-grained task of natural language processing to analyze the sentiment polarity of the targets. To improve the performance of sentiment classification, many approaches proposed various attention mechanisms to capture the important context words of a target. However, previous approaches ignored the significant relatedness of a target's sentiment and its local context. This paper proposes a local context-aware network (LCA-Net), equipped with the local context embedding and local context prediction loss, to strengthen the model by emphasizing the …
Nanjing University（南京大学）；Collaborative Innovation Center of Novel Software Technology and Industrialization（新型软件技术与产业化协同创新中心）
Target-oriented opinion words extraction (TOWE) is a new subtask of ABSA, which aims to extract the corresponding opinion words for a given opinion target in a sentence. Recently, neural network methods have been applied to this task and achieve promising results. However, the difficulty of annotation causes the datasets of TOWE to be insufficient, which heavily limits the performance of neural models. By contrast, abundant review sentiment classification data are easily available at online review sites. These reviews contain substantial latent opinions information and semantic patterns. In th…
Explainable Multimodal Aspect-Based Sentiment Analysis with Dependency-guided Large Language Model
Harbin Institute of Technology（哈尔滨工程大学）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；University of Science and Technology of China（中国科学技术大学）
TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；Tencent（腾讯）；School of Information Science and Technology, Donghua University（东华大学信息科学与技术学院）
ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts
Faculty of Information Science and Engineering, University of Information Technology（信息科学与工程学院，信息科技大学）；Vietnam National University, Ho Chi Minh City（越南国家大学胡志明市分校）
South China Normal University（华南师范大学）；Baidu Inc.（百度公司）
Aspect-based sentiment analysis (ABSA) task is a multi-grained task of natural language processing and consists of two subtasks: aspect term extraction (ATE) and aspect polarity classification (APC). Most of the existing work focuses on the subtask of aspect term polarity inferring and ignores the significance of aspect term extraction. Besides, the existing researches do not pay attention to the research of the Chinese-oriented ABSA task. Based on the local context focus (LCF) mechanism, this paper firstly proposes a multi-task learning model for Chinese-oriented aspect-based sentiment analys…
University of Electronic Science and Technology of China（电子科技大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
Sentiment analysis (SA) is an important research area in cognitive computation-thus in-depth studies of patterns of sentiment analysis are necessary. At present, rich resource data-based SA has been well developed, while the more challenging and practical multi-source unsupervised SA (i.e. a target domain SA by transferring from multiple source domains) is seldom studied. The challenges behind this problem mainly locate in the lack of supervision information, the semantic gaps among domains (i.e., domain shifts), and the loss of knowledge. However, existing methods either lack the distinguisha…
Syntactic parsing, the process of obtaining the internal structure of sentences in natural languages, is a crucial task for artificial intelligence applications that need to extract meaning from natural language text or speech. Sentiment analysis is one example of application for which parsing has recently proven useful. In recent years, there have been significant advances in the accuracy of parsing algorithms. In this article, we perform an empirical, task-oriented evaluation to determine how parsing accuracy influences the performance of a state-of-the-art rule-based sentiment analysis syst…
Tokyo Institute of Technology（东京工业大学）；Microsoft STCA（微软（亚洲）互联网工程院）；Nanjing University（南京大学）；Microsoft Research Asia（微软亚洲研究院）；Westlake University（西湖大学）
Target-oriented Opinion Words Extraction (TOWE) is a fine-grained sentiment analysis task that aims to extract the corresponding opinion words of a given opinion target from the sentence. Recently, deep learning approaches have made remarkable progress on this task. Nevertheless, the TOWE task still suffers from the scarcity of training data due to the expensive data annotation process. Limited labeled data increase the risk of distribution shift between test data and training data. In this paper, we propose exploiting massive unlabeled data to reduce the risk by increasing the exposure of the…
Target-oriented opinion words extraction (TOWE) is a subtask of aspect-based sentiment analysis (ABSA). Given a sentence and an aspect term occurring in the sentence, TOWE extracts the corresponding opinion words for the aspect term. TOWE has two types of instance. In the first type, aspect terms are associated with at least one opinion word, while in the second type, aspect terms do not have corresponding opinion words. However, previous researches trained and evaluated their models with only the first type of instance, resulting in a sample selection bias problem. Specifically, TOWE models w…
University of Sheffield（谢菲尔德大学）；Beihang University（北京航空航天大学）
Target-oriented opinion words extraction (TOWE) (Fan et al., 2019b) is a new subtask of target-oriented sentiment analysis that aims to extract opinion words for a given aspect in text. Current state-of-the-art methods leverage position embeddings to capture the relative position of a word to the target. However, the performance of these methods depends on the ability to incorporate this information into word representations. In this paper, we explore a variety of text encoders based on pretrained word embeddings or language models that leverage part-of-speech and position embeddings, aiming t…
We propose the Vision-and-Augmented-Language Transformer (VAuLT). VAuLT is an extension of the popular Vision-and-Language Transformer (ViLT), and improves performance on vision-and-language (VL) tasks that involve more complex text inputs than image captions while having minimal impact on training and inference efficiency. ViLT, importantly, enables efficient training and inference in VL tasks, achieved by encoding images using a linear projection of patches instead of an object detector. However, it is pretrained on captioning datasets, where the language input is simple, literal, and descri…
PTA: Enhancing Multimodal Sentiment Analysis through Pipelined Prediction and Translation-based Alignment
National University of Defense Technology（国防科技大学）；Hefei University of Technology（合肥工业大学）；Central China Normal University（华中师范大学）；Anhui Agricultural University（安徽农业大学）
Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）
Recent years have witnessed the emerging success of leveraging syntax graphs for the target sentiment classification task. However, we discover that existing syntax-based models suffer from two issues: noisy information aggregation and loss of distant correlations. In this paper, we propose a novel model termed Neural Subgraph Explorer, which (1) reduces the noisy information via pruning target-irrelevant nodes on the syntax graph; (2) introduces beneficial first-order connections between the target and its related words into the obtained graph. Specifically, we design a multi-hop actions scor…
Fairness Metric Design Exploration in Multi-Domain Moral Sentiment Classification using Transformer-Based Models
Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学院）；Department of Computer Science and Technology, SUNY Empire State University（纽约州立大学奥本大学计算机科学与技术系）；Faculty of Computer Studies, Arab Open University, Oman（阿曼开放大学计算机研究学院）
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Accenture Labs（埃森哲实验室）
Movies reflect society and also hold power to transform opinions. Social biases and stereotypes present in movies can cause extensive damage due to their reach. These biases are not always found to be the need of storyline but can creep in as the author's bias. Movie production houses would prefer to ascertain that the bias present in a script is the story's demand. Today, when deep learning models can give human-level accuracy in multiple tasks, having an AI solution to identify the biases present in the script at the writing stage can help them avoid the inconvenience of stalled release, law…
DepFlow: Disentangled Speech Generation to Mitigate Semantic Bias in Depression Detection
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Electrical Engineering and Telecommunications, UNSW（新南威尔士大学电信工程学院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；Center for AI in Medicine (C-AIM), Lee Kong Chian School of Medicine, NTU（人工智能医学中心（C-AIM）、李光耀医学院、南洋理工大学）
