Large language models (LLMs) are increasingly deployed in multilingual, real-world applications with user inputs -- naturally introducing \emph{typographical errors} (typos). Yet most benchmarks assume clean input, leaving the robustness of LLMs to typos across languages largely underexplored. To address this gap, we introduce MulTypo, a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. We evaluate 18 open-source LLMs across three model families and five downstream tasks spanning language inference, multi-choice question answering, mathematical reasoning, and machine translation tasks. Our results show that typos consistently degrade performance, particularly in generative tasks and those requiring reasoning -- while the natural language inference task is comparatively more robust. Instruction tuning improves clean-input performance but may increase brittleness under noise. We also observe language-dependent robustness: high-resource languages are generally more robust than low-resource ones, and translation from English is more robust than translation into English. Our findings underscore the need for noise-aware training and multilingual robustness evaluation. We release a Python package for MulTypo and make the source code publicly available at https://github.com/cisnlp/multypo.
Guardians of Discourse: Evaluating LLMs on Multilingual Offensive Language Detection
School of AI and Advanced Computing, Xi’an Jiaotong-Liverpool University(西交利物浦大学人工智能与先进计算学院);School of Advanced Technology, Xi’an Jiaotong-Liverpool University(西交利物浦大学先进技术学院);College of Computer Science and Technology, Zhejiang University of Technology(浙江工业大学计算机科学与技术学院);Australian Artificial Intelligence Institute, University of Technology Sydney(悉尼科技大学澳大利亚人工智能研究所)
Disentangling Language and Culture for Evaluating Multilingual Large Language Models
Singapore Management University(新加坡管理学院);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院);DAMO Academy, Alibaba Group(阿里集团大模型实验室);Singapore University of Technology and Design(新加坡科技设计大学)
Evaluating Zero-Shot Multilingual Aspect-Based Sentiment Analysis with Large Language Models
Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials, School of Electronic Science and Engineering (School of Microelectronics)(广东量子工程与材料重点实验室,电子科学与工程学院(微电子学院));South China Normal University(华南师范大学);Guangdong Provincial Key Laboratory of Intelligent Information Processing(广东智能信息处理重点实验室);Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Technical University of Munich(慕尼黑技术大学)
Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);Shanghai University(上海大学)
From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
School of Informatics, Xiamen University(厦门大学信息学院);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能学院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室(厦门大学),文化和旅游部)
Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification
AIRI;Universidade de Santiago de Compostela(圣塞巴斯蒂安大学);Technical University of Munich (TUM)(慕尼黑技术大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Skoltech
CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);College of Software, Jilin University(吉林大学软件学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.(中国电信人工智能技术(北京)有限公司星辰AGI实验室);Bilibili Inc.(哔哩哔哩公司);University of Ottawa(渥太华大学);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models
Nanyang Technological University(南洋理工大学);DAMO Academy, Alibaba Group(阿里达摩院);Hupan Lab(虎斑实验室);National University of Singapore(新加坡国立大学);Shanda AI Research Institute(沙达AI研究院)
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团);TU Wien(维也纳技术大学);University of Chinese Academy of Sciences(中国科学院大学)