Question answering platforms, such as Stack Overflow, have impacted substantially how developers search for solutions for their programming problems. The crowd knowledge content available from such platforms has also been used to leverage software development tools. The recent advances on Natural Language Processing, specifically on more powerful language models, have demonstrated ability to enhance text understanding and generation. In this context, we aim at investigating the factors that can influence on the application of such models for understanding source code related data and produce more interactive and intelligent assistants for software development. In this preliminary study, we particularly investigate if a how-to question filter and the level of context in the question may impact the results of a question answering transformer-based model. We suggest that fine-tuning models with corpus based on how-to questions can impact positively in the model and more contextualized questions also induce more objective answers.
HistoRAG: A Citation-Grounded Question Answering Assistant for Teaching with Scanned Local History and Heritage Archives
Mudanjiang Normal University（牡丹江师范学院）；School of Autonomous Driving, Xinwei Institute of Artificial Intelligence（Xinwei人工智能学院自动驾驶学院）；Center for Mathematical Research, Xinwei Institute of Artificial Intelligence（Xinwei人工智能学院数学研究中心）
The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution
City University of Hong Kong (Dongguan)（香港城市大学（东莞））；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology（香港科技大学）
Feedback is critical in education. This Innovative Practice Full Paper reports lessons learned from improving the quality of feedback in a semi-capstone software engineering course, with particular focus on how to deliver productive feedback in large scale during project work. The bachelor-level introduction to software engineering course is taken by about 500 students from eight study programs, organised into 72 project teams. The course aims to educate reflective systems developers. The teaching staff includes 29 teaching assistants as supervisor and product owners for teams. Project teams g…
Cyber-physical systems, such as learning robots and other autonomous systems, employ high-integrity software in their safety-critical control. This software is developed using a range of tools some of which need to be qualified for this purpose according to international standards. In this article, we first evaluate the state of the art of tool qualification for proof assistants, checkers (e.g., model checkers), and generators (e.g., code generators, compilers) by means of a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis. Our focus is on the qualification of tools in the three m…
Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into educational practice, not instinctively defending against it
Newcastle University（纽卡斯尔大学）
针对生成式 AI 编程助手带来的行业变革，本文通过访谈从业者了解实践与挑战，提出将 GAI 融入软件开发教育的愿景及教学建议。
Let the Agent Do It? How Software Practitioners Understand and Make Permission Decisions in Agentic AI Assistants
Chalmers University of Technology（查尔姆斯理工大学）；University of Gothenburg（哥德堡大学）；The University of Melbourne（墨尔本大学）；Eindhoven University of Technology（埃因霍温理工大学）
Virtual Assistant technology is rapidly proliferating to improve productivity in a variety of tasks. While several virtual assistants for everyday tasks are well-known (e.g., Siri, Cortana, Alexa), assistants for specialty tasks such as software engineering are rarer. One key reason software engineering assistants are rare is that very few experimental datasets are available and suitable for training the AI that is the bedrock of current virtual assistants. In this paper, we present a set of Wizard of Oz experiments that we designed to build a dataset for creating a virtual assistant. Our targ…
Understanding the Impact of AI Code Assistants on Security API Usage: An Empirical Study
Centre for Research on Engineering Software Technologies (CREST) & Adelaide University, Australia（工程软件研究技术中心（CREST）及阿德莱德大学，澳大利亚）；Edith Cowan University, Australia（埃德温·考文大学，澳大利亚）；CSIRO’s Data61, Australia（CSIRO数据61，澳大利亚）
Towards Trustworthy AI Software Development Assistance
Technische Universität Darmstadt（德累斯顿技术大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；National Research Center for Applied Cybersecurity ATHENE（应用网络安全国家研究中心ATHENE）
Literature reviews are essential for any researcher trying to keep up to date with the burgeoning software engineering literature. FAST$^2$ is a novel tool for reducing the effort required for conducting literature reviews by assisting the researchers to find the next promising paper to read (among a set of unread papers). This paper describes FAST$^2$ and tests it on four large software engineering literature reviews conducted by Wahono (2015), Hall (2012), Radjenović (2013) and Kitchenham (2017). We find that FAST$^2$ is a faster and robust tool to assist researcher finding relevant SE paper…
Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild
Adelaide University（阿德莱德大学）；University of Victoria（维多利亚大学）；Huazhong University of Science and Technology（华中科技大学）；Singapore Management University（新加坡管理大学）
Fuzzing is one of the fastest growing fields in software testing. The idea behind fuzzing is to check the behavior of software against a large number of randomly generated inputs, trying to cover all interesting parts of the input space, while observing the tested software for anomalous behaviour. One of the biggest challenges facing fuzzer users is how to validate software behavior, and how to improve the quality of oracles used. While mutation analysis is the premier technique for evaluating the quality of software test oracles, mutation score is rarely used as a metric for evaluating fuzzer…
CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants
Brookhaven National Laboratory（布鲁克海文国家实验室）；Stony Brook University（石溪大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Argonne National Laboratory（阿贡国家实验室）
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
In this survey paper, we overview major deep learning methods used in Natural Language Processing (NLP) and source code over the last 35 years. Next, we present a survey of the applications of Artificial Intelligence (AI) for source code, also known as Code Intelligence (CI) and Programming Language Processing (PLP). We survey over 287 publications and present a software-engineering centered taxonomy for CI placing each of the works into one category describing how it best assists the software development cycle. Then, we overview the field of conversational assistants and their applications in…
With the development of artificial intelligence, writing assistants (WAs) are changing the way people interact with text, creating lengthy outputs that can be overwhelming for users. The programming field has long addressed this issue, and Integrated Development Environments (IDEs) have been created for efficient software development, helping programmers reduce the cognitive load. This experience could be employed in the development of WAs. IDEs can also be used to test assumptions about interventions that help people interact with WAs efficiently. Previous works have successfully used self-wr…
AI-Driven Self-Evolving Software: A Promising Path Toward Software Automation
College of AI, Tsinghua University（清华大学人工智能学院）；School of Computer Science, Peking University（北京大学计算机学院）；School of Computer Science and Engineering, Beihang University（北航计算机科学与工程学院）
The research applies AI-driven code assistants to analyze a selection of influential computer code that has shaped modern technology, including email, internet browsing, robotics, and malicious software. The original contribution of this study was to examine half of the most significant code advances in the last 50 years and, in some cases, to provide notable improvements in clarity or performance. The AI-driven code assistant could provide insights into obfuscated code or software lacking explanatory commentary in all cases examined. We generated additional sample problems based on bug correc…
AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?
Key Lab of HCST (PKU), MOE（HCST关键实验室（PKU））；School of Computer Science, Peking University, China（北京大学计算机科学学院）；Columbia University, USA（哥伦比亚大学）；Beijing Jiaotong University, China（北京交通大学）；University of Texas at Dallas, USA（德克萨斯大学达拉斯分校）；Hong Kong University of Science and Technology, Hong Kong SAR, China（香港科技大学）；Fudan University Institute of Systems for Advanced Computing, Shanghai, China（复旦大学先进计算系统研究所）；Simon Fraser University, Canada（西蒙弗雷泽大学）；Shanghai Institute of Systems for Open Computing, Shanghai, China（上海开放计算系统研究所）
Intelligent Personal Assistants (IPAs) are software agents that can perform tasks on behalf of individuals and assist them on many of their daily activities. IPAs capabilities are expanding rapidly due to the recent advances on areas such as natural language processing, machine learning, artificial cognition, and ubiquitous computing, which equip the agents with competences to understand what users say, collect information from everyday ubiquitous devices (e.g., smartphones, wearables, tablets, laptops, cars, household appliances, etc.), learn user preferences, deliver data-driven search resul…
University of Notre Dame（圣母大学）；Clemson University（克莱姆森大学）；Google（谷歌）
This paper targets the problem of speech act detection in conversations about bug repair. We conduct a "Wizard of Oz" experiment with 30 professional programmers, in which the programmers fix bugs for two hours, and use a simulated virtual assistant for help. Then, we use an open coding manual annotation procedure to identify the speech act types in the conversations. Finally, we train and evaluate a supervised learning algorithm to automatically detect the speech act types in the conversations. In 30 two-hour conversations, we made 2459 annotations and uncovered 26 speech act types. Our autom…
