We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation at https://github.com/for-ai/language-confusion.
Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);State Key Laboratory of Computer Science(计算机科学国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学)
LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs
National University of Singapore(新加坡国立大学);VinAI Research(VinAI研究院);Institute of High Performance Computing (IHPC), A*STAR(新加坡科技研究局高性能计算研究所);Salesforce Research(Salesforce研究院);Nanyang Technological University(南洋理工大学)
Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledge
Beijing Institute of Technology, China(北京理工大学);Beijing University of Posts and Telecommunications, China(北京邮电大学);Meituan, China(美团);The Open University, UK(开放大学)
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
Zhejiang University(浙江大学);University of Science and Technology of China(中国科学技术大学);Bytedance(字节跳动);Zhejiang Lab(浙江省实验室);Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence(浙江省医学影像人工智能重点实验室)
Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
University of Maryland(马里兰大学);ELLIS Institute Tübingen(埃利斯研究所蒂宾根分支);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(蒂宾根人工智能中心)
Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);The Chinese University of Hong Kong(香港中文大学);King’s College London(伦敦国王学院);The 30th Research Institute of China Electronics Technology Group Corporation(中国电子科技集团公司第三十研究所)
Can LLM be a Good Path Planner based on Prompt Engineering? Mitigating the Hallucination for Path Planning
College of Computer Science(计算机科学学院);Sichuan Normal University(四川师范大学);School of Information and Software Engineering(信息与软件工程学院);University of Electronic Science and Technology of China(电子科技大学)
Mitigating Hallucinations of Large Language Models in Medical Information Extraction via Contrastive Decoding
University of Science and Technology of China(中国科学技术大学);Jarvis Research Center, Tencent YouTu Lab(腾讯优图实验室贾维斯研究中心);Peking University(北京大学);Westlake University(西湖大学);City University of Hong Kong(香港城市大学)
Truth-Aware Context Selection: Mitigating Hallucinations of Large Language Models Being Misled by Untruthful Contexts
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室)
Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系)
Mitigating Strategy Preference Bias in Emotional Support Conversation via Uncertainty Estimations
Shanghai Institute of Artificial Intelligence for Education(上海人工智能教育研究院);School of Computer Science and Technology(计算机科学与技术学院);School of Psychology and Cognitive Science(心理学与认知科学学院);School of Pharmacy(药学院)