Code LLMs often portray inconsistent program behaviors. Developers typically employ benchmarks to assess Code LLMs, but most benchmarks are hand-crafted, static and do not target consistency property. In this work, we pose the scientific question: how can we automatically discover inconsistent program behaviors in Code LLMs? To address this challenge, we propose an automated consistency testing method, called MUCOCO, which employs semantic-preserving mutation analysis to expose inconsistent behaviors in code LLMs. Given a coding query, MUCOCO automatically transforms its program into semantically equivalent programs (aka mutants) and detects inconsistencies between the mutants and the original program (e.g., different output or test failure). We evaluate MUCOCO using four (4) coding tasks and seven (7) LLMs. Results show that MUCOCO is effective in exposing inconsistency and outperforms the closest baseline (TURBULENCE). About one in seven (15%) inputs generated by MUCOCO exposed inconsistencies. Our work motivates the need to test Code LLMs for consistency property
Beyond Correctness: Exposing LLM-generated Logical Flaws in Reasoning via Multi-step Automated Theorem Proving
Huazhong University of Science and Technology（华中科技大学）；National University of Singapore（新加坡国立大学）；Peking University（北京大学）；Nanyang Technological University（南洋理工大学）
Dependency-Guided Code Generation: Structured Matrix Decomposition and Consistency-Guided Refinement
University of the Chinese Academy of Sciences（中国科学院大学）；ByteDance Inc.（字节跳动公司）；Zhejiang University（浙江大学）；National Taiwan University（台湾国立大学）；Peking University（北京大学）；Alibaba Group（阿里巴巴集团）；Tsinghua University（清华大学）；Liaoning Technical University（辽宁技术大学）
