This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations.
I run as fast as a rabbit, can you? A Multilingual Simile Dialogue Dataset
School of Information Science, Beijing Language and Culture University(北京语言大学信息科学学院);School of Computer Science, Harbin Institute of Technology(哈尔滨工业大学计算机科学学院)
FFE-Hallu:Hallucinations in Fixed Figurative Expressions:Benchmark of Idioms and Proverbs in the Persian Language
Tehran Institute for Advanced Studies, Khatam University, Iran(德黑兰高级研究 institute,Khatam 大学,伊朗);School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran(电气与计算机工程学院,工程学院,德黑兰大学,德黑兰,伊朗)
It's not Rocket Science : Interpreting Figurative Language in Narratives
Columbia University(哥伦比亚大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);University of British Columbia(不列颠哥伦比亚大学)
When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Thailand(泰国拉差班国王理工大学信息科技学院)
Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages
Mohamed bin Zayed University of Artificial Intelligence(莫扎德大学人工智能大学);University of Hamburg(汉堡大学);Manipal University(曼印大学);IIIT Hyderabad(海得拉尔IIIT);University of Science and Technology of Hanoi(河内科学技术大学);Universitas Islam Indonesia(印尼伊斯兰大学);Princeton University(普林斯顿大学)
Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference
Indiana University Bloomington(印第安纳大学伯明顿分校);Brigham Young University(杨百翰大学);Northwestern University(西北大学);UNC Chapel Hill(北卡罗来纳大学教堂山分校);Allen Institute for AI(艾伦人工智能研究所);Shanghai Jiaotong Univ.(上海交通大学)
LMs stand their Ground: Investigating the Effect of Embodiment in Figurative Language Interpretation by Language Models
Ludwig-Maximilians-University (LMU)(路德维希-马克西米利安大学(慕尼黑大学));Institute for Information and Language Processing (CIS)(信息与语言处理研究所);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
Cross-Lingual Steering for Figurative Language Generation
Department of Computer Science, University of Cincinnati(卡内基梅隆大学计算机科学系);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机科学学院);Independent Researcher(独立研究者)
MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages
ellamind;Leibniz University Hannover(莱布尼茨汉诺威大学);University of Helsinki(赫尔辛基大学);University of Turku(图尔库大学);ELLIS Institute Tübingen(ELLIS 图宾根研究所);Prior Labs;University of Oslo(奥斯陆大学);Prompsit Language Engineering(Prompsit 语言工程公司);AI Sweden;Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico(高等技术学院);TransPerfect;Charles University(查理大学);Ontocord;LAION;Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)(于利希超级计算中心 (JSC), 于利希研究中心 (FZJ))