Recently, large language models (LLM) based generative AI has been gaining momentum for their impressive high-quality performances in multiple domains, particularly after the release of the ChatGPT. Many believe that they have the potential to perform general-purpose problem-solving in software development and replace human software developers. Nevertheless, there are in a lack of serious investigation into the capability of these LLM techniques in fulfilling software development tasks. In a controlled 2 x 2 between-subject experiment with 109 participants, we examined whether and to what degree working with ChatGPT was helpful in the coding task and typical software development task and how people work with ChatGPT. We found that while ChatGPT performed well in solving simple coding problems, its performance in supporting typical software development tasks was not that good. We also observed the interactions between participants and ChatGPT and found the relations between the interactions and the outcomes. Our study thus provides first-hand insights into using ChatGPT to fulfill software engineering tasks with real-world developers and motivates the need for novel interaction mechanisms that help developers effectively work with large language models to achieve desired outcomes.
From Documents to Spans: Span-Centric Learning for LLM-based Explainable Medical Coding
School of Biomedical Engineering, Division of Life Sciences and Medicine, USTC（生物医学工程学院，生命科学与医学系，中国科学技术大学）；MIRACLE Center, Suzhou Institute for Advance Research, USTC（MIRACLE中心，苏州先进研究院，中国科学技术大学）；Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology（江苏省多模态数字孪生技术重点实验室）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（精密与智能化学国家重点实验室）
The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models
Department of IT IIIT Allahabad（信息技术系，印度阿拉哈巴德IIIT）；Department of ECE IIIT Allahabad（电子工程系，印度阿拉哈巴德IIIT）；Department of CSE IIT Kanpur（计算机科学与工程系，印度坎purIIT）
The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers
MIT-IBM Watson AI Lab（MIT-IBM Watson人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）；Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）；IBM Research（IBM研究院）；University of California, Berkeley（加州大学伯克利分校）
Crystal: Illuminating LLM Abilities on Language and Code
Carnegie Mellon University（卡内基梅隆大学）；Cerebras Systems（赛布拉斯系统公司）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
From Code-Centric to Intent-Centric Software Engineering: A Reflexive Thematic Analysis of Generative AI, Agentic Systems, and Engineering Accountability
SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning
Department of Computer Science, Columbia University, New York, USA（哥伦比亚大学计算机科学系）；Computer Science Department, University of California, Los Angeles, California, USA（加州大学洛杉矶分校计算机科学系）
Rosetta Memory: Adaptive Memory for Cross-LLM Agents
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Weixin, Tencent（腾讯微信）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Chinese Information Processing Laboratory（中文信息处理实验室）；State Key Laboratory of Computer Science（计算机科学国家重点实验室）；Key Laboratory of System Software, Chinese Academy of Sciences（中国科学院系统软件重点实验室）
BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Monash University（墨尔本大学）；CSIRO’s Data61（CSIRO的数据61）；Purdue University（普渡大学）；Independent（独立）；HKUST (Guangzhou)（香港科技大学（广州））；UCSD（加州大学圣地亚哥分校）；UVA（弗吉尼亚大学）；CNRS, France（法国国家科学研究中心）；IBM；Cisco（思科）；Comenius University in Bratislava（布拉提斯拉瓦康门纽斯大学）；University of Notre Dame（Notre Dame大学）；Uber；Tano Labs（Tano实验室）；NUS（国立大学新加坡）；Institute of Automation, CAS（中国科学院自动化研究所）；Tencent AI Lab（腾讯AI实验室）；University of Washington（华盛顿大学）；Nevsky Collective（Nevsky集体）；ETH Zurich（苏黎世联邦理工学院）；Detomo Inc（Detomo公司）；University of Oxford（牛津大学）；UIUC（伊利诺伊大学香槟分校）；Google（谷歌）；NVIDIA；Singapore Management University（新加坡管理学院）；ServiceNow Research（ServiceNow研究）；Hugging Face
Specula: Scaling formal specifications for autonomous model checking of system code
Nanjing University（南京大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Research Asia（微软亚洲研究院）；University of British Columbia（不列颠哥伦比亚大学）
AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering
University of Houston（休斯顿大学）；New York University（纽约大学）；Texas A&M University（德克萨斯农工大学）；UCSD（加州大学圣地亚哥分校）；Worcester Polytechnic Institute（伍斯特理工学院）；Oregon State University（俄勒冈州立大学）
SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
University of Notre Dame（圣母大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Utah（犹他大学）；University of Nebraska–Lincoln（内布拉斯加大学林肯分校）；The Ohio State University（俄亥俄州立大学）；Argonne National Laboratory（阿贡国家实验室）；Anthropic PBC
Agentsway -- Software Development Methodology for AI Agents-based Teams
Old Dominion University（旧 Dominion 大学）；Florida International University（佛罗里达国际大学）；University of Sri Jayewardenepura（Sri Jayewardenepura 大学）；Deloitte & Touche LLP；AnaletIQ；Nanyang Technological University（南洋理工大学）；University of Colombo（科伦坡大学）
