Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification
Dalian University of Technology（大连理工大学）；School of Foreign Languages（外国语学院）；School of Software（软件学院）；University of Macau（澳门大学）；Faculty of Business Administration（工商管理学院）；Kansai University（关西大学）；Faculty of Business and Commerce（商学部）；University of New South Wales（新南威尔士大学）；School of Minerals and Energy Resources Engineering（矿产与能源资源工程学院）；RMIT University（皇家墨尔本理工大学）；Centre for Educational Innovation and Quality（教育创新与质量中心）；School of Computing Technologies（计算技术学院）
Metaphors play a crucial role in human communication, yet their comprehension remains a significant challenge for natural language processing (NLP) due to the cognitive complexity involved. According to Conceptual Metaphor Theory (CMT), metaphors map a target domain onto a source domain, and understanding this mapping is essential for grasping the nature of metaphors. While existing NLP research has focused on tasks like metaphor detection and sentiment analysis of metaphorical expressions, there has been limited attention to the intricate process of identifying the mappings between source and target domains. Moreover, non-English multimodal metaphor resources remain largely neglected in the literature, hindering a deeper understanding of the key elements involved in metaphor interpretation. To address this gap, we developed a Chinese multimodal metaphor advertisement dataset (namely CM3D) that includes annotations of specific target and source domains. This dataset aims to foster further research into metaphor comprehension, particularly in non-English languages. Furthermore, we propose a Chain-of-Thought (CoT) Prompting-based Metaphor Mapping Identification Model (CPMMIM), which simulates the human cognitive process for identifying these mappings. Drawing inspiration from CoT reasoning and Bi-Level Optimization (BLO), we treat the task as a hierarchical identification problem, enabling more accurate and interpretable metaphor mapping. Our experimental results demonstrate the effectiveness of CPMMIM, highlighting its potential for advancing metaphor comprehension in NLP. Our dataset and code are both publicly available to encourage further advancements in this field.
MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors
Centre for Linguistics, Humanities, Leiden University（莱顿大学人文学院语言学中心）；LIACS, Leiden University（莱顿大学LIACS）；BDS, Leiden University Medical Centre（莱顿大学医学中心BDS）
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
University of International Relations（国际关系学院）；Kedge Business School（凯致商学院）；Peking University（北京大学）；Tsinghua University（清华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Capital Normal University（首都师范大学）
MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Institute of Infectious Disease and Biosecurity, Fudan University（复旦大学上海传染病与生物安全研究院）；Shanghai Health Development Research Center (Shanghai Medical Information Center)（上海市卫生和健康发展研究中心（上海市医学科学技术情报研究所））；Imperial College London（伦敦帝国理工学院）
CMDAG: A Chinese Metaphor Dataset with Annotated Grounds as CoT for Boosting Metaphor Generation
University of California, San Diego（加利福尼亚大学圣迭戈分校）；Massachusetts Institute of Technology（麻省理工学院）；HKUST（香港科技大学）；University of Manchester（曼彻斯特大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Waterloo（滑铁卢大学）；Multimodal Art Projection Research Community（多模态艺术投影研究社区）
Enhancing Multimodal Large Language Models for Ancient Chinese Character Evolution Analysis via Glyph-Driven Fine-Tuning
College of Computer Science and Technology, Jilin University, China（吉林大学计算机科学与技术学院）；Key Laboratory of Ancient Chinese Script, Culture Relics and Artificial Intelligence, Jilin University, China（吉林大学古文字、文物与人工智能重点实验室）；School of Artificial Intelligence, Jilin University, China（吉林大学人工智能学院）；School of Archaeology, Jilin University, China（吉林大学考古学院）
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Multimodal Art Projection Research Community（多模态艺术投影研究共同体）；Hong Kong University of Science and Technology（香港科技大学）；University of Waterloo（滑铁卢大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peking University（北京大学）；The Hong Kong Polytechnic University（香港理工大学）；Waseda University（早稻田大学）；University of Manchester（曼彻斯特大学）
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Freedom AI；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Guangdong University of Technology（广东工业大学）
EmoMeta: A Multimodal Dataset for Fine-grained Emotion Classification in Chinese Metaphors
School of Software（软件学院）；Dalian University of Technology（大连理工大学）；School of Foreign Languages and School of Software（外语学院和软件学院）；Faculty of Business Administration（商学院）；University of Macau（澳门大学）；School of Computing Technologies（计算技术学院）；RMIT University（皇家墨尔本理工大学）
A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks
DJCAD, University of Dundee, United Kingdom（邓迪大学DJCAD部门）；ARG-tech, SSEN, University of Dundee, United Kingdom（邓迪大学）；School of Computer Science, University of Birmingham, United Kingdom（伯明翰大学计算机科学学院）
University of Surrey（萨里大学）；University of Sheffield（谢菲尔德大学）
Metaphor generation is a challenging task which can impact many downstream tasks such as improving user satisfaction with dialogue systems and story generation. This paper tackles the problem of Chinese nominal metaphor generation by introducing a multitask metaphor generation framework with self-training and metaphor identification mechanisms. Self-training addresses the data scarcity issue of metaphor datasets. That is, instead of solely relying on labelled metaphor datasets which are usually small in size, self-training helps identify potential metaphors from a large-scale unlabelled corpus…
University of Helsinki（赫尔辛基大学）；Bose Corporation（博士公司）
We present the first openly available multimodal metaphor annotated corpus. The corpus consists of videos including audio and subtitles that have been annotated by experts. Furthermore, we present a method for detecting metaphors in the new dataset based on the textual content of the videos. The method achieves a high F1-score (62\%) for metaphorical labels. We also experiment with other modalities and multimodal methods; however, these methods did not out-perform the text-based model. In our error analysis, we do identify that there are cases where video could help in disambiguating metaphors…
Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Tsinghua University（清华大学）；Zhihu Inc.（知乎公司）；ModelBest Inc.（模贝斯特公司）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Renmin University of China（中国人民大学）
BenCao: An Instruction-Tuned Large Language Model for Traditional Chinese Medicine
Community Health Service Center Shanghai Pudong New Area（上海浦东新区社区卫生服务中心）；School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine（上海中医药大学针灸推拿学院）
本文提出基于 ChatGPT 的中医药多模态助手 BenCao，通过自然语言指令微调、知识库、思维链模拟、专家反馈及外部 API 整合，提升诊断、药材识别与体质分类性能并实现部署。
Towards Comprehensive Detection of Chinese Harmful Memes
School of Computer Science and Technology（计算机科学与技术学院）；School of Foreign Languages（外国语学院）；Key Laboratory of Social Computing and Cognitive Intelligence（社会计算与认知智能重点实验室）；Dalian University of Technology（大连理工大学）
WordCraft: Scaffolding the Keyword Method for L2 Vocabulary Learning with Multimodal LLMs
School of Information Science and Technology, ShanghaiTech University（信息科学与技术学院，上海科技大学）；School of Creativity and Art, ShanghaiTech University（创意与艺术学院，上海科技大学）；Shanghai Fengxian Dai Wen Middle School（上海奉贤戴文中学）
University of Ottawa（渥太华大学）；American University（美利坚大学）
We apply a large multilingual language model (BLOOM-176B) in open-ended generation of Chinese song lyrics, and evaluate the resulting lyrics for coherence and creativity using human reviewers. We find that current computational metrics for evaluating large language model outputs (MAUVE) have limitations in evaluation of creative writing. We note that the human concept of creativity requires lyrics to be both comprehensible and distinctive -- and that humans assess certain types of machine-generated lyrics to score more highly than real lyrics by popular artists. Inspired by the inherently mult…
Beijing Institute of Technology（北京理工大学）；Tencent Cloud Xiaowei（腾讯云小微）
Chinese Spell Checking (CSC) aims to detect and correct erroneous characters for user-generated text in the Chinese language. Most of the Chinese spelling errors are misused semantically, phonetically or graphically similar characters. Previous attempts noticed this phenomenon and try to use the similarity for this task. However, these methods use either heuristics or handcrafted confusion sets to predict the correct character. In this paper, we propose a Chinese spell checker called ReaLiSe, by directly leveraging the multimodal information of the Chinese characters. The ReaLiSe model tackles…
Multi-Granular Multimodal Clue Fusion for Meme Understanding
School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院）；National University of Singapore（新加坡国立大学）；North China Institute of Computing Technology（华北计算技术研究所）
Yangzhou University（扬州大学）；Hefei University of Technology（合肥工业大学）
Idioms, are a kind of idiomatic expression in Chinese, most of which consist of four Chinese characters. Due to the properties of non-compositionality and metaphorical meaning, Chinese Idioms are hard to be understood by children and non-native speakers. This study proposes a novel task, denoted as Chinese Idiom Paraphrasing (CIP). CIP aims to rephrase idioms-included sentences to non-idiomatic ones under the premise of preserving the original sentence's meaning. Since the sentences without idioms are easier handled by Chinese NLP systems, CIP can be used to pre-process Chinese datasets, there…
With the development of the Internet, more and more people get accustomed to online shopping. When communicating with customer service, users may express their requirements by means of text, images, and videos, which precipitates the need for understanding these multimodal information for automatic customer service systems. Images usually act as discriminators for product models, or indicators of product failures, which play important roles in the E-commerce scenario. On the other hand, detailed information provided by the images is limited, and typically, customer service systems cannot under…
In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding perfor…
CFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；Peking University School of Computer Science（北京大学计算机科学学院）；Peking University School of Software and Microelectronics（北京大学软件与微电子学院）
