Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.
Can ChatGPT Really Understand Modern Chinese Poetry?
NLP;CT Lab, Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门科技大学CT实验室);Department of Portuguese, Faculty of Arts and Humanities, University of Macau(人文学院葡萄牙语系,澳门科技大学)
ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models
Institute for Software Technology, German Aerospace Center (DLR)(德国航空航天中心(DLR)软件技术研究所);Technical University Darmstadt(达姆施塔特工业大学);Centre for Cognitive Science, TU Darmstadt(达姆施塔特工业大学认知科学中心);Hessian Center for AI (hessian.AI)(黑森人工智能中心(hessian.AI))
Differentiate ChatGPT-generated and Human-written Medical Texts
South China University of Technology(华南理工大学);University of Georgia(佐治亚大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital and Harvard Medical School(马萨诸塞总医院和哈佛医学院)
ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection
Ingenuity Labs Research Institute(创新实验室研究所);Queen’s University(女王大学);Smith School of Business(史密斯商学院);Department of Electrical and Computer Engineering(电气与计算机工程系)
Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model
Carnegie Mellon University(卡内基梅隆大学);LMU Munich(慕尼黑大学);University of Oxford(牛津大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Allen Institute for AI(艾伦人工智能研究所);IIT Delhi(德里印度理工学院)
When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学);LG AI Research(LG人工智能研究院);University of Illinois Chicago(芝加哥大学伊利诺伊分校);University of Michigan(密歇根大学)
Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering(陕西省大数据知识工程重点实验室);National Engineering Lab for Big Data Analytics(国家大数据分析工程实验室);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院)
Model Attribution in LLM-Generated Disinformation: A Domain Generalization Approach with Supervised Contrastive Learning
School of Computing;AI Arizona State University Tempe, AZ;Department of Computer Science Illinois Institute of Technology Chicago, IL;Department of Computer Science Emory University Atlanta, GA
NegativePrompt: Leveraging Psychology for Large Language Models Enhancement via Negative Emotional Stimuli
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Institute of Software, CAS(中国科学院软件研究所);Microsoft Research Asia(微软亚洲研究院);Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University(吉林大学符号计算与知识工程重点实验室);International Center of Future Science, Jilin University(吉林大学未来科学国际合作联合实验室)
PointLLM: Empowering Large Language Models to Understand Point Clouds
The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Centre for Perceptual and Interactive Intelligence(感知与交互智能中心)
Emulating Human Cognitive Processes for Expert-Level Medical Question-Answering with Large Language Models
Dell Medical School at the University of Texas(德克萨斯大学戴尔医学院);William Beaumont University Hospital(威廉·博蒙特大学医院);Austin State Hospital(奥斯汀州立医院);Institute of Mental Health, University of Nottingham(诺丁汉大学精神卫生研究所);Government Medical College and Hospital, Chandigarh(昌迪加尔政府医学院及医院);University of Augsburg(奥格斯堡大学);McGovern Medical School and The University of Texas Health Science Center(麦戈文医学院与德克萨斯大学健康科学中心);Sheikh Shakhbout Medical City(谢赫·沙赫布特医疗城);Dr Akbar Niazi Teaching Hospital(阿克巴尔·尼亚兹博士教学医院);Centro Médico ABC(ABC医疗中心);Houston Regional Gastroenterology Institute(休斯顿地区胃肠病学研究所);NeuroReef Labs(NeuroReef实验室)
University of Applied Sciences HTW Berlin(柏林应用技术大学);Riga Technical University(里加技术大学);Uppsala University(乌普萨拉大学);Masaryk University(马萨里克大学);Universidad de Monterrey(蒙特雷大学);Queen Mary University of London(伦敦玛丽女王大学);University of Leeds(利兹大学)
Large Language Models (LLMs) and Generative AI in Cybersecurity and Privacy: A Survey of Dual-Use Risks, AI-Generated Malware, Explainability, and Defensive Strategies
Virelya Intelligence Research Labs(Virelya智能研究实验室);Google(谷歌)
How Quantum Is the Advantage? A Fair, Calibration- and Noise-Aware Benchmark and Attribution Audit of Quantum Machine Learning for Network Intrusion Detection
A Survey on Post-training of Large Language Models
Huazhong University of Science and Technology(华中科技大学);Lehigh University(莱斯大学);The University of Hong Kong(香港大学);Jilin University(吉林大学);Southern University of Science and Technology(南方科技大学);Worcester Polytechnic Institute(沃思堡理工学院);LinkedIn Corporation(领英公司);Squirrel Ai Learning;University of Georgia(佐治亚大学);Duke University(杜克大学);Michigan State University(密歇根州立大学);Salesforce Research(Salesforce研究);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Microsoft Research(微软研究院)