Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only benchmarks. However, current research lacks such benchmarks for multimodal visio-linguistic settings. This work fills this gap by introducing M5, the first comprehensive benchmark designed to evaluate LMMs on diverse vision-language tasks within a multilingual and multicultural context. M5 includes eight datasets covering five tasks and $41$ languages, with a focus on underrepresented languages and culturally diverse images. Furthermore, we introduce two novel datasets, M5-VGR and M5-VLOD, including a new Visio-Linguistic Outlier Detection task, in which all evaluated open-source models fail to significantly surpass the random baseline. Through extensive evaluation and analyses, we highlight substantial task-agnostic performance disparities between high- and low-resource languages. Moreover, we show that larger models do not necessarily outperform smaller ones in a multilingual setting.
M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed人工智能大学);Ubiquitous Knowledge Processing Lab(ubiquitous知识处理实验室);Department of Computer Science, TU Darmstadt(TU Darmstadt计算机科学系);National Research Center for Applied Cybersecurity ATHENE(应用网络安全国家研究中心ATHENE);Department of Electrical Engineering, KU Leuven(KU Leuven电气工程系);Department of Computer Science, KU Leuven(KU Leuven计算机科学系)
SemEval-2026 Task 9: Detecting Multilingual, Multicultural and Multievent Online Polarization
Macquarie University(麦考瑞大学);University of Hamburg(汉堡大学);Zayed University(扎耶德大学);HKBK College of Engineering(HKBK工程学院);University of Turin(都灵大学);aequa-tech;University of Pretoria(比勒陀利亚大学);Bielefeld University(比勒费尔德大学);Jagiellonian University(雅盖隆大学);Bahir Dar University(巴赫达尔大学);AIRI;KFU(喀山联邦大学);HSE University(高等经济大学);Virginia Tech(弗吉尼亚理工大学);IIT Delhi(印度理工学院德里分校);ABV-IIITM(ABV-印度信息技术与管理学院);Qatar Computing Research Institute(卡塔尔计算研究所);Hamad Bin Khalifa University(哈马德·本·哈利法大学);Language Understanding Lab., Myanmar(缅甸语言理解实验室);AMD Silo AI;Lahore University of Management Sciences(拉合尔管理科学大学);Maseno University(马塞诺大学);Centrum Wiskunde & Informatica(数学与计算机科学中心);Bayero University Kano(卡诺巴耶罗大学);Northeastern University(东北大学);Imperial College London(伦敦帝国学院)
Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability
Google DeepMind(谷歌DeepMind);University of Alabama in Huntsville(阿拉巴马大学亨茨维尔分校);Google(谷歌);University of Missouri Columbia(密苏里大学哥伦比亚分校);Chunghwa Telecom Laboratories(春木电信实验室);University of Southern California(南加州大学);Polytechnique Montreal(蒙特利尔理工学院);Nutanix;ThinkEvolve Labs(ThinkEvolve实验室);UCL(伦敦大学学院);Infocomm Media Development Authority(信息通信媒体发展局);Monark Health(Monark健康);Seoul National University(首尔国立大学);Microsoft(微软);Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), Indian Institute of Technology Madras(负责任人工智能中心(CeRAI)、瓦达威人工智能学校(WSAI)、印度理工学院马德拉斯分校);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Microsoft Research India(微软印度研究院);Stanford University(斯坦福大学);Argonne National Laboratory(阿贡国家实验室);University of Oxford(牛津大学);KAIST(韩国科学技术院);Yonsei University(延世大学);Amazon(亚马逊);UIUC(伊利诺伊大学香槟分校);Rochester Institute of Technology(罗切斯特理工学院);Xenoscube Inc.(Xenoscube公司);Korea AI Safety Institute (K-AISI)(韩国人工智能安全研究所(K-AISI));MLCommons;CommonGround;Artifex Labs(Artifex实验室)
Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports
Indian Institute of Technology Patna(印度理工学院帕纳布分校);Sardar Patel Institute of Technology(萨达尔·帕特尔技术学院);Universitas Gadjah Mada(加查马大学);King Mongkut’s Institute of Technology Ladkrabang(拉差班国王技术学院);Shenzhen Technology University(深圳技术大学);Université de Toulouse(图卢兹大学)
UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment
University of Bath(巴斯大学);Cardiff University(卡迪夫大学);National University Philippines(菲律宾国家大学);Bielefeld University(比勒菲尔德大学);University of Exeter(埃克塞特大学);University of Gothenburg(哥德堡大学);UCLouvain(列日大学);MBZUAI(马克斯·普朗克智能系统研究所);Brigham Young University(Brigham Young 大学);INESC-ID Lisboa(里斯本 INESC-ID);Instituto Universitário de Lisboa (ISCTE-IUL)(里斯本大学学院(ISCTE-IUL));Universitat Pompeu Fabra(庞培法华大学);National Research Council, Canada(加拿大国家研究委员会);King Abdulaziz University(国王阿卜杜勒-阿齐兹大学)
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Alibaba Group(阿里巴巴集团);Beijing Language and Culture University(北京语言大学);LMU Munich(慕尼黑大学);ELLIS Institute Finland(芬兰ELLIS研究所);University of Turku(图尔库大学);IBM Research AI, UAE(阿联酋IBM人工智能研究);MBZUAI;Harbin Institute of Technology, Shenzhen(深圳哈尔滨工业大学);Southern University of Science and Technology(南方科技大学)
ParCourE: A Parallel Corpus Explorer for a Massively Multilingual Corpus
Center for Information and Language Processing (CIS), LMU Munich(慕尼黑大学信息与语言处理中心(CIS));Research Center Deutscher Sprachatlas, Philipps University Marburg(马尔堡菲利普大学德国语言地图集研究中心)
MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset
Instituto de Telecomunicações(电信研究所);Facebook AI(脸书人工智能研究院);IQT Labs(IQT实验室);University of Sheffield(谢菲尔德大学);Imperial College London(伦敦帝国学院);Unbabel(Unbabel公司);Johns Hopkins University(约翰霍普金斯大学);University of Wolverhampton(伍尔弗汉普顿大学)
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
Capital One;SEACrowd;NAIST;ITB;UofT;KAIST;JAIST;Cardiff University;Independent;HKUST;SMU;Masakhane;University of Lagos;HK PolyU;Boston University;Columbia University;MBZUAI;Monash University;UCSB;Tokyo Tech;AI Singapore;NICT;Cohere;McGill;MILA;Canada CIFAR AI Chair;Ontario Tech
AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic
Saarland University(萨尔兰大学);AIMS AMMI;Resonance AI4D Lab(Resonance AI4D实验室);Addis Ababa University(亚的斯亚贝巴大学);HILCOE School of Computer Science and Technology(HILCOE计算机科学与技术学院);University of Hamburg(汉堡大学)
PRHLT Research Center, Universitat Politècnica de València(PRHLT研究中心,瓦伦西亚理工大学);AUSY R&D(奥西研发中心);IRIT, CNRS, Université de Toulouse(图卢兹大学计算机研究所,法国国家科学研究中心)
INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages
McGill University(麦吉尔大学);Mila, Quebec AI Institute(米拉魁北克人工智能研究所);Saarland University(萨尔大学);SADiLaR(南非数字语言资源中心);University of Toronto(多伦多大学);OntarioTech University(安大略理工大学);Princeton University(普林斯顿大学);Makerere University(马凯雷雷大学);L3S Research Center(L3S研究中心);Imperial College London(伦敦帝国学院);Universidad de Deusto(德乌斯托大学);Dakar American University Of Science and Technology(达喀尔美国科技大学)
University of Washington(华盛顿大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Carnegie Mellon University(卡内基梅隆大学);University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(矢量人工智能研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
Nilekani Centre at AI4Bharat(AI4Bharat 奈尔肯中心);IIT Madras(印度理工学院马德拉斯分校);New York University(纽约大学);Microsoft(微软);National Institute of Information and Communications Technology, Kyoto, Japan(京都国立信息与通信技术研究所)
Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models
Data & Knowledge Engineering, Heinrich Heine University(海因里希·海因大学数据与知识工程系);Center for Language and Cognition (CLCG), University of Groningen(格罗宁根大学语言与认知中心)