To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution. Our NL-Code pairs are harvested from StackOverflow forums to encourage natural and practical coding queries. Moreover, ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian. ODEX unveils intriguing behavioral differences among top-performing code language models (LM). While CODEX achieves better overall results, CODEGEN improves effectively via scaling -- CODEGEN 6.1B performs comparably with CODEX 12B. Both models show substantial gaps between open and closed domains, but CODEGEN gaps tend to decrease with model size while CODEX gaps increase. We release ODEX to facilitate research into open-domain problems for the code generation community.
CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
School of Computer Science, Peking University(北京大学计算机科学系);Tongyi Lab, Alibaba Group(阿里集团通义实验室);School of Computer Science, Wuhan University(武汉大学计算机科学系)
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团);TU Wien(维也纳技术大学);University of Chinese Academy of Sciences(中国科学院大学)
Execution-based Evaluation for Data Science Code Generation Models
Beihang University(北京航空航天大学);Microsoft Research Redmond(微软研究院雷德蒙德分院);Microsoft Research Asia(微软亚洲研究院);Microsoft(微软公司);Hong Kong University of Science and Technology(香港科技大学);Toronto University(多伦多大学)
Natural Language to Code Translation with Execution
Meta AI;Toyota Technological Institute at Chicago(芝加哥丰田技术研究院);Carnegie Mellon University(卡内基梅隆大学);University of Washington(华盛顿大学)
在大型程序语料库上预训练的代码生成模型在将自然语言转换为代码方面已经显示出巨大的成功(Chen et al.,2021; Austin等人,2021年; Li等人,2022年,除其他外)。虽然这些模型没有明确地结合程序语义(即,执行结果),在培训期间,他们能够对许多问题产生正确的解决方案。然而,选择一个正确的程序从每个问题的生成集仍然具有挑战性。在这项工作中,我们引入了基于执行结果的最小贝叶斯风险解码(MBR-EXEC)的程序选择,并表明它提高了自然语言到代码任务的预训练代码模型的Few-Shot性能。我们选择输出程序从一个产生的候选人集边缘化的程序实现,共享相同的语义。由于精确等价是棘手的,我们执行每个程序上的少量测试输入近似语义等价。跨数据集,执行或模拟执行显著优于不涉及程序语义的方法。我们发现,MBR-EXEC一贯改善所有执行不知情的选择方法,这表明它是一种有效的方法,自然语言的代码翻译。我们在github.com/facebookresearch/mbr-exec上开源我们的代码,在dl.fbaipublicfiles.com/mbr-exec/mbr-exec-release.zip上开源我们的数据
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
Multimodal Art Projection Research Community(多模态艺术投影研究共同体);University of Waterloo(滑铁卢大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);HKUST(香港科技大学);IN.AI Research(IN.AI研究机构)
Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs
Xi’an Jiaotong University(西安交通大学);Institute of AI for Industries, Chinese Academy of Sciences(中国科学院人工智能产业研究院);The University of Tokyo(东京大学);University of Alberta(阿尔伯塔大学)
ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
Institute of Trustworthy Embodied AI(可信具身AI研究院);Shanghai Innovation Institution(上海创新机构);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身AI重点实验室);College of Computer Science and Artificial Intelligence(计算机科学与人工智能学院);OpenMOSS;Wuhan University(武汉大学);Nanjing University(南京大学);Jilin University(吉林大学)
BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Monash University(墨尔本大学);CSIRO’s Data61(CSIRO的数据61);Purdue University(普渡大学);Independent(独立);HKUST (Guangzhou)(香港科技大学(广州));UCSD(加州大学圣地亚哥分校);UVA(弗吉尼亚大学);CNRS, France(法国国家科学研究中心);IBM;Cisco(思科);Comenius University in Bratislava(布拉提斯拉瓦康门纽斯大学);University of Notre Dame(Notre Dame大学);Uber;Tano Labs(Tano实验室);NUS(国立大学新加坡);Institute of Automation, CAS(中国科学院自动化研究所);Tencent AI Lab(腾讯AI实验室);University of Washington(华盛顿大学);Nevsky Collective(Nevsky集体);ETH Zurich(苏黎世联邦理工学院);Detomo Inc(Detomo公司);University of Oxford(牛津大学);UIUC(伊利诺伊大学香槟分校);Google(谷歌);NVIDIA;Singapore Management University(新加坡管理学院);ServiceNow Research(ServiceNow研究);Hugging Face
SpaceTime Programming: Live and Omniscient Exploration of Code and Execution
Université de Rennes, IRISA(雷恩大学,IRISA);Université de Rennes, CNRS, IRISA(雷恩大学,CNRS,IRISA);Université de Rennes, Inria, CNRS, IRISA(雷恩大学,Inria,CNRS,IRISA)
Tree-of-Code: A Hybrid Approach for Robust Complex Task Planning and Execution
Global Innovation Exchange Institution, Tsinghua University(清华大学全球创新学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Baidu, Inc.(百度公司)
CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
Hong Kong Baptist University(香港浸会大学);UC Santa Barbara(加州大学圣巴巴拉分校);UC Berkeley(加州大学伯克利分校);Alibaba Group(阿里巴巴集团);The University of Tokyo(东京大学);University of Alberta(阿尔伯塔大学)
RedCode: Risky Code Execution and Generation Benchmark for Code Agents
University of Chicago(芝加哥大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Lapis Labs(拉皮斯实验室);Virginia Tech(弗吉尼亚理工大学);Microsoft Research(微软研究院);University of California Berkeley(加州大学伯克利分校)
GenAI-based test case generation and execution in SDV platform
Mercedes-Benz AG, Germany(梅赛德斯-奔驰集团,德国);Technical University of Munich, Germany(慕尼黑技术大学,德国);Ferdinand-Steinbeis-Institut der Steinbeis-Stiftung, Germany(施坦贝格基金会费尔迪南-斯坦贝格研究所,德国)