Demystifying Issues, Causes and Solutions in LLM Open-Source Projects
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science & Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning, Central China Normal University(华中师范大学计算机学院);School of Computing Technologies, RMIT University(RMIT大学计算技术学院)
With the advancements of Large Language Models (LLMs), an increasing number of open-source software projects are using LLMs as their core functional component. Although research and practice on LLMs are capturing considerable interest, no dedicated studies explored the challenges faced by practitioners of LLM open-source projects, the causes of these challenges, and potential solutions. To fill this research gap, we conducted an empirical study to understand the issues that practitioners encounter when developing and using LLM open-source software, the possible causes of these issues, and potential solutions. We collected all closed issues from 15 LLM open-source projects and labelled issues that met our requirements. We then randomly selected 994 issues from the labelled issues as the sample for data extraction and analysis to understand the prevalent issues, their underlying causes, and potential solutions. Our study results show that (1) Model Issue is the most common issue faced by practitioners, (2) Model Problem, Configuration and Connection Problem, and Feature and Method Problem are identified as the most frequent causes of the issues, and (3) Optimize Model is the predominant solution to the issues. Based on the study results, we provide implications for practitioners and researchers of LLM open-source projects.
Demystifying the Silence of Correctness Bugs in PyTorch Compiler
The Hong Kong University of Science and Technology(香港科技大学);Guangzhou HKUST Fok Ying Tung Research Institute(广州香港科技大学霍英东研究院);Carnegie Mellon University(卡内基梅隆大学)
LLM-based Vulnerability Detection at Project Scale: An Empirical Study
School of Computer Software, College of Intelligence and Computing, Tianjin University(天津大学计算机科学与技术学院);International Joint Institute of Tianjin University(天津大学国际联合学院);Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, School of Computer Science, Peking University(北京大学计算机科学学院,教育部高可信软件技术重点实验室)
ARFT-Transformer: Modeling Metric Dependencies for Cross-Project Aging-Related Bug Prediction
College of Information Engineering, Capital Normal University, Beijing, China(信息工程学院,首都师范大学,北京,中国);Suzhou Aerospace Information Research Institute, Suzhou, China(苏州航天信息研究所,苏州,中国);School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, Beijing, China(机械电子控制工程学院,北京交通大学,北京,中国);Beijing Institute of Computer Technology and Applications, Beijing, China(北京计算机技术与应用研究所,北京,中国);School of Automation Science and Electrical Engineering, Beihang University, Beijing, China(自动化科学与电气工程学院,北航,北京,中国)