arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.13620cs.SE
收藏

编程语言混淆:当代码LLM无法保持语言一致

Programming Language Confusion: When Code LLMs Can't Keep their Languages Straight

University of Luxembourg(卢森堡大学)

Micheline Bénédicte Moumoula, Serge Lionel Nikiema, Abdoul Kader Kabore, Jacques Klein, Tegawendé F. Bissyande

AI总结:

研究揭示代码LLM在编程语言混淆问题上的系统性错误,提出通过显式语言关键词缓解,并倡导标准化评估以提升多语言代码生成系统的可靠性。

摘要:

大型语言模型(LLMs)在软件工程任务上已取得最先进的性能,从代码生成到翻译。然而,我们识别并系统地评估了一个关键故障模式:编程语言混淆(PLC)——尽管有明确的指示,仍然生成意外语言的代码。通过在六个多语言数据集(LiveCodeBench、BabelCode变体、HumanEval-XL和McEval)上评估10种流行的LLM,我们证明PLC普遍存在,一些专门的模型表现出最高的混淆率。我们的分析揭示,PLC不是随机噪声,而是反映了系统性模式:模型即使偏离语言规范,也会生成语法上有效的代码。这种行为会产生不同的语言迁移模式,最显著的是对Python的强烈默认倾向以及在语法相似的语言对之间系统性转移(例如C#/Java)。这些迁移反映了从训练数据中学习的统计偏好,而不是目标导向的推理。我们证明,显式的语言关键词提供了最有效的缓解方法,而自然语言指令对模型行为影响有限。此外,模型量化——尽管对实际部署至关重要——会显著加剧PLC并降低复杂任务中的语法稳定性。我们的发现强调,语言准确性应被视为代码LLM的核心评估维度。我们倡导标准化的基准和带有明确语言约束的提示格式,以实现更可靠的评估,并促进稳健、多语言代码生成系统的发展。

英文摘要:

Large Language Models (LLMs) have achieved state-of-the-art performance across software engineering tasks, from code generation to translation. However, we identify and systematically evaluate a critical failure mode: Programming Language Confusion (PLC) -- the generation of code in unintended languages despite explicit instructions. Through evaluation of 10 popular LLMs across six multilingual datasets (LiveCodeBench, BabelCode variants, HumanEval-XL, and McEval), we demonstrate that PLC is pervasive, with some specialized models exhibiting the highest confusion rates. Our analysis reveals that PLC is not random noise but reflects systematic patterns: models consistently generate syntactically valid code even when it deviates from language specifications. This behavior produces distinct language migration patterns, most notably a strong default to Python and systematic shifts between syntactically similar language pairs (e.g., C#/Java). These migrations reflect statistical preferences learned from training data rather than goal-directed reasoning. We demonstrate that explicit language keywords provide the most effective mitigation, while natural language instructions have limited influence on model behavior. Furthermore, model quantization -- though essential for practical deployment -- significantly amplifies PLC and degrades syntactic stability in complex tasks. Our findings underscore that language fidelity should be treated as a core evaluation dimension for code LLMs. We advocate for standardized benchmarks and prompt formats with explicit language constraints to enable more reliable assessment and foster the development of robust, multilingual code generation systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑