Programming-by-Examples (PBE) aims to generate an algorithm from input-output examples. Such systems are practically and theoretically important: from an end-user perspective, they are deployed to millions of people, and from an AI perspective, PBE corresponds to a very general form of few-shot inductive inference. Given the success of Large Language Models (LLMs) in code-generation tasks, we investigate here the extent to which LLMs can be said to have "solved" PBE. We experiment on classic domains such as lists and strings, and an uncommon graphics programming domain not well represented in typical pretraining data. We find that pretrained models are not effective at PBE, but that they can be fine-tuned for much higher performance, provided the test problems are in-distribution. We analyze empirically what causes these models to succeed and fail, and take steps toward understanding how to achieve better out-of-distribution generalization. Collectively these results suggest that LLMs make strong progress toward solving the typical suite of PBE tasks, potentially increasing the flexibility and applicability of PBE systems, while also identifying ways in which LLMs still fall short.
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
New York University(纽约大学);Princeton University(普林斯顿大学);University of California San Diego(加利福尼亚大学圣地亚哥分校);University of Washington(华盛顿大学);Canyon Crest Academy(峡谷克雷斯特学院);University of Waterloo(滑铁卢大学);McGill University(麦吉尔大学)
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团);TU Wien(维也纳技术大学);University of Chinese Academy of Sciences(中国科学院大学)
Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems
Shanghai Qi Zhi Institute(上海启智研究院);ShanghaiTech University(上海科技大学);Wuhan University(武汉大学);Fuzhou University(福州大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学);Nanjing University(南京大学);Beijing University of Posts and Telecommunications(北京邮电大学);Peking University(北京大学)
Automatic Programming: Large Language Models and Beyond
Chinese University of Hong Kong(香港中文大学);Columbia University(哥伦比亚大学);National University of Singapore(新加坡国立大学);Concordia University(康考迪亚大学);University of Melbourne(墨尔本大学)
InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);Key Laboratory of Cyberspace Security, Ministry of Education(教育部网络空间安全重点实验室);Hubei Key Laboratory of Distributed System Security(湖北省分布式系统安全重点实验室)
Fudan University(复旦大学);Peking University(北京大学);Nanyang Technological University(南洋理工大学);Huazhong University of Science and Technology(华中科技大学);Harbin Institute of Technology(哈尔滨工业大学);Meituan Inc(美团公司)
astra-langchain4j: Experiences Combining LLMs and Agent Programming
School of Computer Science, University College Dublin, Ireland(都柏林大学计算机科学学院);School of Computer Science, University of St.Gallen, St. Gallen, Switzerland(圣加尔登大学计算机科学学院)
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Hefei University of Technology(合肥工业大学);School of Artificial Intelligence and Data Science, University of Science and Technology of China(中国科学技术大学人工智能与数据科学学院);University of Toronto(多伦多大学)