Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code summarization. Despite their promise, researchers must still be careful as numerous intricate factors can influence the outcomes of experiments involving LLMs. This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings. In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns. The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.
Privacy-preservation for sensitive data has become a challenging issue in cloud computing. Threat modeling as a part of requirements engineering in secure software development provides a structured approach for identifying attacks and proposing countermeasures against the exploitation of vulnerabilities in a system . This paper describes an extension of Cloud Privacy Threat Modeling (CPTM) methodology for privacy threat modeling in relation to processing sensitive data in cloud computing environments. It describes the modeling methodology that involved applying Method Engineering to specify ch…
As we are continuously depending on information technology applications by adopting electronic channels and software applications for our business, online transaction and communication, software security is increasingly becoming a necessity and more advanced concern. Both the functional and non-functional requirements are important and provide the necessary needs at the early phases of the software development process, specifically in the requirement phase. The aim of this research is to identify security threats early in the software development process to help the requirement engineer elicit…
In requirements engineering (RE), knowledge is mainly communicated via written specifications. This practice is cumbersome due to its low communication richness and effectiveness. In contrast, videos can transfer knowledge more richly and effectively. However, video is still a neglected medium in RE. We investigate if software professionals perceive video as a medium that can contribute to RE. We focus on their attitudes towards video as a medium in RE including its strengths, weaknesses, opportunities, and threats. We conducted a survey to explore these attitudes with a questionnaire. 64 out …
Machine learning (ML) has been widely used in the literature to automate software engineering tasks. However, ML outcomes may be sensitive to randomization in data sampling mechanisms and learning procedures. To understand whether and how researchers in SE address these threats, we surveyed 45 recent papers related to three predictive tasks: defect prediction (DP), predictive mutation testing (PMT), and code smell detection (CSD). We found that less than 50% of the surveyed papers address the threats related to randomized data sampling (via multiple repetitions); only 8% of the papers address …
Background. There are some publications in software engineering research that aim at guiding researchers in assessing validity threats to their studies. Still, many researchers fail to address many aspects of validity that are essential to quantitative research on human factors. Goal. This paper has the goal of triggering a change of mindset in what types of studies are the most valuable to the behavioral software engineering field, and also provide more details of what construct validity is. Method. The approach is based on psychological test theory and draws upon methods used in psychology i…
Mitigating Omitted Variable Bias in Empirical Software Engineering
Software Institute, USI Università della Svizzera italiana（瑞士意大利大学软件研究所）；Chalmers University of Technology and University of Gothenburg（查尔姆斯理工大学和哥德堡大学）；Stellenbosch Institute for Advanced Study (STIAS)（斯坦伦布什高级研究院）
For a qualitative system sound security practices must be a crucial part throughout the entire software lifecycle. Furthermore, agile software development has paved the way for overcoming the problems faced by developers during traditional development process. In the given paper we are using an Agile Security Framework that is compatible with practices of agile processes and inherit in it the benefits of security engineering activities in the form of risk assessment and threat prioritization. One of the most popular techniques to deal with ever growing risks associated with security threats is…
Future of Software Engineering Research: The SIGSOFT Perspective
University of Sannio（桑尼奥大学）；University of Auckland（奥克兰大学）；University of Toronto（多伦多大学）；Carnegie Mellon University（卡内基梅隆大学）；Singapore Management University（新加坡管理大学）；Microsoft, Inc.（微软公司）；University of California, Irvine（加州大学尔湾分校）
[Context] In software engineering research, emphasis is given to sound evaluations of new approaches. While industry surveys or industrial case studies are preferred to evaluate industrial applicability, controlled experiments with student participants are commonly used to determine measurements such as effectiveness and efficiency of a proposed approach. [Objectives] In this paper, we elaborate on the current state of the art of controlled experiments using student participants. As student participants are commonly only reluctantly accepted in scientific communities and threats regarding the …
Software vulnerabilities in access control models can represent a serious threat in a system. In fact, OWASP lists broken access control as number 5 in severity among the top 10 vulnerabilities. In this paper, we study the permission model of an emerging Smart-Home platform, SmartThings, and explore an approach that detects privilege escalation in its permission model. Our approach is based on Model Driven Engineering (MDE) in addition to static analysis. This approach allows for better coverage of privilege escalation detection than static analysis alone, and takes advantage of analyzing free…
Background. Software Engineering (SE) researchers extensively perform experiments with human subjects. Well-defined samples are required to ensure external validity. Samples are selected \textit{purposely} or by \textit{convenience}, limiting the generalizability of results. Objective. We aim to depict the current status of participants selection in empirical SE, identifying the main threats and how they are mitigated. We draft a robust approach to participants' selection. Method. We reviewed existing participants' selection guidelines in SE, and performed a preliminary literature review to fi…
Journalists, public policy analysts, and economists have called attention to the growing importance that high-performance and scientific computing have to national security and industrial leadership. As computing continues to power scientific advances in virtually every discipline, so too does it improve our economic productivity and quality of life. The increasing social, political, and economic importance of research software, however, has also brought the question of software security to the fore. Just as unintentional software errors can threaten the integrity of scientific studies, malici…
Context: The Evidence-Based Software Engineering (EBSE) paradigm and the planning phase of a systematic literature review. Objective: A protocol to do a systematic literature review with detailed information about the processes suggested by several guidelines in the field of evidence-based software engineering. Method: An analisys of recent systematic literature reviews published in world leading journals, plus the use of two renowned guidelines and a textbook to sinthetise a formal plan (the protocol). Results: The validated protocol Conclusions: We found that most of the published systematic…
Human-AI Collaboration in Requirements Engineering: Evidence of the Negative Effect of LLMs on Requirements Inspection
CNR–ISTI, Pisa, Italy（意大利比萨国家研究委员会信息与通信技术研究所）；Chalmers University of Technology, Gothenburg, Sweden（瑞典哥德堡查尔姆斯理工大学）；Monash University, Melbourne, Australia（澳大利亚墨尔本蒙纳士大学）；University of Florence, Florence, Italy（意大利佛罗伦萨大学）；University of Duisburg-Essen（德国杜伊斯堡-埃森大学）；Trinity College Dublin, Dublin, Ireland（爱尔兰都柏林三一学院）
Background: Software Engineering regularly views communication between project participants as a tool for solving various problems in software development. Objective: Formulate research questions in areas related to CHASE. Method: A day-long discussion of five participants at the in-person day of the 15th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE 2022) on May 23rd 2022. Results: It is not rare in industrial SE projects that communication is not just a tool or technique to be applied but also represents a resource, which, when lacking, threatens pr…
Generative AI and Empirical Software Engineering: A Paradigm Shift
School of Computing（计算学院）；Information Systems Singapore Management University Singapore, Singapore（信息系统新加坡管理大学新加坡）；Department of Computer Science University of Victoria Victoria, Canada（计算机科学系维多利亚大学加拿大）
Social Coding Platforms (SCPs) like GitHub have become central to modern software engineering thanks to their collaborative and version-control features. Like in mainstream Online Social Networks (OSNs) such as Facebook, users of SCPs are subjected to privacy attacks and threats given the high amounts of personal and project-related data available in their profiles and software repositories. However, unlike in OSNs, the privacy concerns and practices of SCP users have not been extensively explored nor documented in the current literature. In this work, we present the preliminary results of an …
