Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to generate a natural language sentence for a multimodal social post (an image as well as its caption) to explain why it contains sarcasm. Although the existing pioneer study has achieved great success with the BART backbone, it overlooks the gap between the visual feature space and the decoder semantic space, the object-level metadata of the image, as well as the potential external knowledge. To solve these limitations, in this work, we propose a novel mulTi-source sEmantic grAph-based Multimodal sarcasm explanation scheme, named TEAM. In particular, TEAM extracts the object-level semantic meta-data instead of the traditional global visual features from the input image. Meanwhile, TEAM resorts to ConceptNet to obtain the external related knowledge concepts for the input text and the extracted object meta-data. Thereafter, TEAM introduces a multi-source semantic graph that comprehensively characterize the multi-source (i.e., caption, object meta-data, external knowledge) semantic relations to facilitate the sarcasm reasoning. Extensive experiments on a public released dataset MORE verify the superiority of our model over cutting-edge methods.
CFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark
National Key Laboratory for Multimedia Information Processing(多媒体信息处理国家重点实验室);Peking University School of Computer Science(北京大学计算机科学学院);Peking University School of Software and Microelectronics(北京大学软件与微电子学院)
MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detection
Max Planck Institute for Informatics, Saarland Informatics Campus(马克斯·普朗克研究所信息学研究所,萨尔兰州信息学校园);Saarland University(萨尔兰州大学);The University of Edinburgh(爱丁堡大学);Samsung AI Center, Cambridge(三星人工智能中心,剑桥)
Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal Sarcasm Detection
MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Beihang University(北京航空航天大学)
StanceFlip: A Comprehensive Multi-Dimensional Benchmark for Multimodal Conversational Stance Flipping Forecasting
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件学院);Harbin Institute of Technology Shenzhen(哈尔滨工业大学(深圳));City University of Macau(澳门城市大学)
Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models
Anhui Polytechnic University (AHPU)(安徽工程大学);Shanghai University (SHU)(上海大学);Shanghai Jiao Tong University (SJTU)(上海交通大学);Sichuan University (SCU)(四川大学)
Explaining (Sarcastic) Utterances to Enhance Affect Understanding in Multimodal Dialogues
Indraprastha Institute of Information Technology Delhi(德里英德拉普拉斯塔信息技术学院);University of Maryland, College Park(马里兰大学帕克分校);Indian Institute of Technology Delhi(印度理工学院德里分校)
GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection
State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)(人工智能安全国家重点实验室,计算技术研究所(ICT),中国科学院(CAS));University of Chinese Academy of Sciences, CAS(中国科学院大学);Pengcheng Laboratory(鹏城实验室);Shenzhen Stock Exchange(深圳证券交易所)
SarcasmMiner: A Dual-Track Post-Training Framework for Robust Audio-Visual Sarcasm Reasoning
Speech Technology Lab, University of Groningen, The Netherlands(格罗宁根大学语音技术实验室,荷兰);Center for Language and Cognition, University of Groningen, The Netherlands(格罗宁根大学语言与认知中心,荷兰)
Multi-View Incongruity Learning for Multimodal Sarcasm Detection
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);State Key Laboratory of Software Development Environment, Beihang University(北京航空航天大学软件开发环境国家重点实验室);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机学院);Department of Computer Science, University of Illinois at Chicago(伊利诺伊大学芝加哥分校计算机系)
NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving
ARC Training Centre for Automated Vehicles in Rural and Remote Regions (AVR3)(澳大利亚农村及偏远地区自动驾驶车辆ARC培训中心);Université Gustave Eiffel(Gustave Eiffel大学);Queensland University of Technology (QUT)(昆士兰科技大学)
Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue
Shandong University(山东大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);Shandong Jianzhu University(山东建筑大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model
University of Liverpool, UK(利物浦大学);Nanyang Technological University, SG(南洋理工大学);WMG, University of Warwick(沃里克大学工程学院);The Chinese University of Hong Kong, Shenzhen, Guangdong, China(香港中文大学(深圳))
SemIRNet: A Semantic Irony Recognition Network for Multimodal Sarcasm Detection
University of New South Wales(新南威尔士大学);University of Sydney(悉尼大学);University of Pennsylvania(宾夕法尼亚大学);University of Southern California(南加州大学);Kazan Federal University(卡扎那夫联邦大学)