Large language models have demonstrated great potential to assist programmers in generating code. For such human-AI pair programming scenarios, we empirically demonstrate that while generated code is most often evaluated in terms of their functional correctness (i.e., whether generations pass available unit tests), correctness does not fully capture (e.g., may underestimate) the productivity gains these models may provide. Through a user study with N = 49 experienced programmers, we show that while correctness captures high-value generations, programmers still rate code that fails unit tests as valuable if it reduces the overall effort needed to complete a coding task. Finally, we propose a hybrid metric that combines functional correctness and syntactic similarity and show that it achieves a 14% stronger correlation with value and can therefore better represent real-world gains when evaluating and comparing models.
Reward Difference Optimization For Sample Reweighting In Offline RLHF
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；The Chinese University of Hong Kong（香港中文大学）；New Jersey Institute of Technology（新泽西理工学院）
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
National Institute of Informatics (NII)（国立信息学研究所）；University of Amsterdam（阿姆斯特丹大学）；University of Tokyo（东京大学）；Hitotsubashi University（一桥大学）；Centrum Wiskunde & Informatica (CWI)（荷兰数学与计算机科学研究中心）；Utrecht University（乌得勒支大学）
Aligning Large Language Models with Searcher Preferences
School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；Xiaohongshu Inc.（小红书公司）；Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）人工智能研究所）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
