大语言模型能否理解、推理并生成混合语言文本？
Can Large Language Models Understand, Reason About, and Generate Code-Switched Text?
AI总结：
本文通过CodeMixQA基准测试评估大语言模型在理解、推理和生成混合语言文本方面的能力，揭示了模型在混合语言环境中的局限性，并提供了改进多语言LLMs的见解。
摘要：
混合语言现象在多语言交流中普遍存在，但大语言模型（LLMs）在混合语言环境中的鲁棒性仍不够清楚。在本工作中，我们对LLMs在理解、推理和生成混合语言文本方面的能力进行了全面评估。我们引入了CodeMixQA这一新的基准测试，包含高质量的人类注释，由16种多样的平行混合语言语对组成，涵盖多个地理区域和混合语言模式，并包含原始脚本及其转写形式。利用此基准测试，我们分析了LLMs在混合语言问答任务中的推理行为，揭示了模型如何处理和推理混合语言输入。我们进一步对LLMs生成的合成混合语言文本进行了系统评估，重点关注自然性和语义忠实性，并揭示了当前生成能力的关键限制。我们的发现揭示了在混合语言条件下推理和生成的持续挑战，并为构建更健壮的多语言LLMs提供了可行的见解。我们已将数据集和代码作为开源发布。
英文摘要：
Code-switching is a pervasive phenomenon in multilingual communication, yet the robustness of large language models (LLMs) in mixed-language settings remains insufficiently understood. In this work, we present a comprehensive evaluation of LLM capabilities in understanding, reasoning over, and generating code-switched text. We introduce CodeMixQA a novel benchmark with high-quality human annotations, comprising 16 diverse parallel code-switched language-pair variants that span multiple geographic regions and code-switching patterns, and include both original scripts and their transliterated forms. Using this benchmark, we analyze the reasoning behavior of LLMs on code-switched question-answering tasks, shedding light on how models process and reason over mixed-language inputs. We further conduct a systematic evaluation of LLM-generated synthetic code-switched text, focusing on both naturalness and semantic fidelity, and uncover key limitations in current generation capabilities. Our findings reveal persistent challenges in both reasoning and generation under code-switching conditions and provide actionable insights for building more robust multilingual LLMs. We release the dataset and code as open source.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
