Despite the effectiveness of large language models (LLMs) for code generation, they often output incorrect code. One reason is that model output probabilities are often not well-correlated with correctness, and reflect only the final output of the generation process. Inspired by findings that LLMs internally encode concepts like truthfulness, this paper explores if LLMs similarly represent code correctness. Specifically, we identify a correctness representation inside LLMs by contrasting the hidden states between pairs of correct and incorrect code for the same programming tasks. By experimenting on four LLMs, we show that exploiting this extracted correctness representation outperforms standard log-likelihood ranking, as well as verbalized model confidence. Furthermore, we explore how this internal correctness signal can be used to select higher-quality code samples, without requiring test execution. Ultimately, this work demonstrates how leveraging internal representations can enhance code generation systems and make LLMs more reliable, thus improving confidence in automatically generated code.
CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphs
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Edinburgh(爱丁堡大学);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);IQuest Research(IQuest研究)
Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Chinese Information Processing Laboratory(中文信息处理实验室);State Key Laboratory of Computer Science(计算机科学国家重点实验室);Key Laboratory of System Software, Chinese Academy of Sciences(中国科学院系统软件重点实验室)
Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment
Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Beijing Institute for General Artificial Intelligence(北京一般人工智能研究所)
Beijing University of Posts and Telecommunications(北京邮电大学);School of Artificial Intelligence and Data Science, University of International Business and Economics(国际经济贸易大学人工智能与数据科学学院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院)
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
2 State Key Laboratory of Complex \& Critical Software Environment (SKLCCSE) School of Computer Science;Engineering, Beihang University, Beijing, China;3 School of Software, Beihang University, Beijing, China;4 School of Computer Science;Technology, Shandong University, Qingdao, China;5 Huawei Cloud Computing Technologies Co., Ltd, China
ReCatcher: Towards LLMs Regression Testing for Code Generation
Huawei Distributed Scheduling and Data Engine Lab(华为分布式调度与数据引擎实验室);Huawei Distributed Scheduling(华为分布式调度);Data Engine Lab Montreal Canada(数据引擎实验室蒙特利尔加拿大);Data Engine Lab(数据引擎实验室)
Demystifying the Silence of Correctness Bugs in PyTorch Compiler
The Hong Kong University of Science and Technology(香港科技大学);Guangzhou HKUST Fok Ying Tung Research Institute(广州香港科技大学霍英东研究院);Carnegie Mellon University(卡内基梅隆大学)
IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection
Jheronimus Academy of Data Science(Jheronimus数据科学学院);Tilburg University(蒂尔堡大学);Eindhoven University of Technology(埃因霍温理工大学);University of Sannio(萨诺尼大学)
Predicting LLM Correctness in Prosthodontics Using Metadata and Hallucination Signals
1 Department of Data Science, Monash University Indonesia, Tangerang, Indonesia;2 Independent Researcher;3 Department of Prosthodontics, University of Pittsburgh, Pittsburgh, Pennsylvania;4 Department of Restorative Sciences, University of North Carolina Adams School of Dentistry, Chapel Hill, North Carolina;5 Department of Computer Science, Boston University, Boston, Massachusetts
Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
Department of Computer Engineering, Tarsus University(塔鲁斯大学计算机工程系);School of Computing and Augmented Intelligence (SCAI), Arizona State University (ASU)(计算与增强智能学院(SCAI),亚利桑那州立大学(ASU));HumaConn AI Consulting(HumaConn AI咨询)