How Does Naming Affect LLMs on Code Analysis Tasks?
College of Information Science and Technology, The Pennsylvania State University(宾夕法尼亚州立大学信息科学与技术学院);Heinz College, Carnegie Mellon University(卡内基梅隆大学海因茨学院)
Zhilong Wang, Lan Zhang, Chen Cao, Nanqing Luo, Xinzhi Luo, Peng Liu
备注:
3 Table, 8 figures
发表信息:
Journal of Software Engineering and Applications. Vol.17 No.11, November 2024
The Large Language Models (LLMs), such as GPT and BERT, were proposed for natural language processing (NLP) and have shown promising results as general-purpose language models. An increasing number of industry professionals and researchers are adopting LLMs for program analysis tasks. However, one significant difference between programming languages and natural languages is that a programmer has the flexibility to assign any names to variables, methods, and functions in the program, whereas a natural language writer does not. Intuitively, the quality of naming in a program affects the performance of LLMs in program analysis tasks. This paper investigates how naming affects LLMs on code analysis tasks. Specifically, we create a set of datasets with code containing nonsense or misleading names for variables, methods, and functions, respectively. We then use well-trained models (CodeBERT) to perform code analysis tasks on these datasets. The experimental results show that naming has a significant impact on the performance of code analysis tasks based on LLMs, indicating that code representation learning based on LLMs heavily relies on well-defined names in code. Additionally, we conduct a case study on some special code analysis tasks using GPT, providing further insights.
Can Graph Descriptive Order Affect Solving Graph Problems with LLMs?
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of California, Merced(加州大学默塞德分校);SAIDS, University of Science and Technology of China(中国科学技术大学SAIDS);Tsinghua University(清华大学)
When LLMs Team Up: The Emergence of Collaborative Affective Computing
Department of Computing, Hong Kong Polytechnic University(香港理工大学计算机系);School of Data Science, Lingnan University(岭南大学数据科学学院);School of Computer Science and the Data Science Institute, University of Technology Sydney(悉尼大学计算机科学学院和数据科学研究所);Education University of Hong Kong(香港教育大学)
Do Persona-Infused LLMs Affect Performance in a Strategic Reasoning Game?
Bellini College of Artificial Intelligence, Cybersecurity and Computing(人工智能、网络安全与计算学院);University of South Florida(佛罗里达州立大学);Information Directorate(信息 Directorate);Air Force Research Library(空军研究图书馆)
Evaluating the Use of LLMs for Documentation to Code Traceability
Data-driven Analysis of Software (DAS) Lab Department of Computer Science \& Software Engineering Concordia University Montréal QC Canada;Concordia University
code_transformed: The Influence of Large Language Models on Code
Huazhong University of Science and Technology(华中科技大学);École normale supérieure - Université PSL(巴黎高等师范学院-巴黎大学);Laboratoire LATTICE(LATTICE实验室);International School for Advanced Studies (SISSA)(国际先进研究学院(SISSA))
Affective-ROPTester: Capability and Bias Analysis of LLMs in Predicting Retinopathy of Prematurity
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机与数据科学学院);Department of Ophthalmology, Huizhou First Hospital(惠州市第一人民医院眼科部);school of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);School of Education, Shanghai Jiao Tong University(上海交通大学教育学院);School of Computer Science, University of St Andrews(圣安德鲁大学计算机科学学院);Shenzhen Eye Hospital, Shenzhen Eye Medical Center, Southern Medical University(深圳眼医院、深圳眼科医院、南方医科大学)
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学院,北京大学);School of EECS, Peking University(电子工程系,北京大学);Huawei Noah’s Ark Lab(华为诺亚实验室);Center for Machine Learning Research, Peking University(机器学习研究中心,北京大学)
What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks
Shandong University(山东大学);University of Electronic Science and Technology of China(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
LLMs Should Incorporate Explicit Mechanisms for Human Empathy
School of Computer Science, Hangzhou Dianzi University(杭州电子科技大学计算机学院);School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)智能科学与工程学院)
Can "AI" Be a Doctor? A Study of Empathy, Readability, and Alignment in Clinical LLMs
Department of Electrical Engineering and Information Technology(电子工程与信息科技系);University of Naples Federico II(那不勒斯费德里科二世大学);Department of Translational Medical Sciences(转化医学科学系);University of Campania ”Luigi Vanvitelli”(坎帕尼亚“路易吉·范维蒂利”大学);Department of Computer Science, McCormick School of Engineering and Applied Science(计算机科学系,麦科马克工程与应用科学学院);Northwestern University(西北大学)
Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?
Northeastern University, Boston, MA, USA(东北大学,波士顿,马萨诸塞州,美国);The University of Texas MD Anderson Cancer Center, Houston, Texas, USA(德克萨斯大学MD安德森癌症中心,休斯顿,德克萨斯州,美国);King’s College London, London, UK(伦敦国王学院,伦敦,英国);The University of Texas at Austin, Austin, Texas, USA(德克萨斯大学奥斯汀分校,奥斯汀,德克萨斯州,美国)
How should the advent of large language models affect the practice of science?
Max Planck Institute for Biological Cybernetics(马克斯·普朗克生物控制论研究所);Helmholtz Center for Computational Health, Munich(慕尼黑亥姆霍兹计算健康中心);University of Tübingen(图宾根大学);University of California Santa Barbara(加州大学圣巴巴拉分校);Eötvös Loránd University(厄特沃什·罗兰大学);University of Washington(华盛顿大学);Health and Medical University, Potsdam(波茨坦健康与医科大学);Max-Planck-Institute for Human Development(马克斯·普朗克人类发展研究所);University of Basel(巴塞尔大学);Rutgers University, New Brunswick(罗格斯大学新布朗斯维克分校);Indiana University, Bloomington(印第安纳大学伯明顿分校);Harvard University(哈佛大学);University of Zurich(苏黎世大学);University of Milano-Bicocca(米兰比可卡大学);Google DeepMind(谷歌DeepMind);University College London(伦敦大学学院)
Huazhong University of Science and Technology(华中科技大学);École Normale Supérieure (ENS) – Université Paris Sciences et Lettres (PSL)(巴黎高等师范学院(ENS)–巴黎文理研究大学(PSL));International School for Advanced Studies (SISSA)(国际高等研究学院(SISSA))
Catch Me if You Search: When Contextual Web Search Results Affect the Detection of Hallucinations
The Pennsylvania State University(宾夕法尼亚州立大学);Department of Communication & Center for Trustworthy Artificial Intelligence (CTAI)(通信系与可信人工智能中心);Seoul National University(首尔国立大学)