Assessing the Code Clone Detection Capability of Large Language Models
School of Computer Science, University of Galway（戈尔韦大学计算机科学学院）；CRT-AI, the Science Foundation Ireland Centre for Research Training in Artificial Intelligence（爱尔兰科学基金会人工智能研究培训中心（CRT-AI））；Lero, the Science Foundation Ireland Research Centre for Software（爱尔兰科学基金会软件研究中心（Lero））
This study aims to assess the performance of two advanced Large Language Models (LLMs), GPT-3.5 and GPT-4, in the task of code clone detection. The evaluation involves testing the models on a variety of code pairs of different clone types and levels of similarity, sourced from two datasets: BigCloneBench (human-made) and GPTCloneBench (LLM-generated). Findings from the study indicate that GPT-4 consistently surpasses GPT-3.5 across all clone types. A correlation was observed between the GPTs' accuracy at identifying code clones and code similarity, with both GPT models exhibiting low effectiveness in detecting the most complex Type-4 code clones. Additionally, GPT models demonstrate a higher performance identifying code clones in LLM-generated code compared to humans-generated code. However, they do not reach impressive accuracy. These results emphasize the imperative for ongoing enhancements in LLM capabilities, particularly in the recognition of code clones and in mitigating their predisposition towards self-generated code clones--which is likely to become an issue as software engineers are more numerous to leverage LLM-enabled code generation and code refactoring tools.
A Systematic Approach for Assessing Large Language Models' Test Case Generation Capability
R. B. Annis School of Engineering（R.B.安尼斯工程学院）；E. S. Witchger School of Engineering（E.S.威彻格工程学院）；University of Indianapolis（印第安纳波利斯大学）；Marian University（玛丽安大学）
ElectriQ: A Benchmark for Assessing the Response Capability of Large Language Models in Power Marketing
Systems Engineering Institute, Xi’an Jiaotong University（系统工程研究所，西安交通大学）；State Key Laboratory of Multiphase Flow in Power Engineering, School of Energy and Power Engineering, Xi’an Jiaotong University（电力工程多相流国家重点实验室，能源与动力工程学院，西安交通大学）；School of Electronic Science and Engineering, Xi'an Jiaotong University（电子科学与工程学院，西安交通大学）；Organizational Management Department, School of Management, Xi’an Jiaotong University（组织管理部，管理学院，西安交通大学）
PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Software clone detection identifies similar code snippets. It has been an active research topic that attracts extensive attention over the last two decades. In recent years, machine learning (ML) based detectors, especially deep learning-based ones, have demonstrated impressive capability on clone detection. It seems that this longstanding problem has already been tamed owing to the advances in ML techniques. In this work, we would like to challenge the robustness of the recent ML-based clone detectors through code semantic-preserving transformations. We first utilize fifteen simple code trans…
CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
Department of Informatics（信息学院）；King’s College London（伦敦国王学院）；Institute of Psychology（心理学研究所）；University of Tartu（塔尔图大学）；Department of Psychology（心理学系）；Stanford University（斯坦福大学）；The Alan Turing Institute（艾伦·图灵研究所）
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Multimodal Art Projection Research Community（多模态艺术投影研究社区）；University of Waterloo（滑铁卢大学）；HKUST（香港科技大学）；University of Manchester（曼彻斯特大学）；Tongji University（同济大学）；Vector Institute（矢量研究所）
A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis
The University of Sydney（悉尼大学）；Tencent AI Lab, Tencent（腾讯AI实验室）；Guangzhou University of Chinese Medicine（广州中医药大学）；The University of Wollongong（伍伦贡大学）；The University of Adelaide（阿德莱德大学）
CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Process
Ludwig Maximilian University of Munich（慕尼黑路德维希-马克西米利安大学）；Munich Research Center, Huawei Technologies（华为技术有限公司慕尼黑研究中心）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）
From Code to Compliance: Assessing ChatGPT's Utility in Designing an Accessible Webpage -- A Case Study
Department of Computer Science, Norwegian University of Science and Technology（计算机科学系，挪威科学技术大学）；Department of Design, Norwegian University of Science and Technology（设计系，挪威科学技术大学）
EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）
Designing usable products is important to provide a competitive edge through user satisfaction. A first step to establish or improve a usability process is to perform a process assessment. As this may be costly, an alternative for organizations seeking for lighter assessments, especially small organizations, may be self-assessments. They can be carried out by an organization on its own to assess the capability of its process. Although there are specific assessment methods to assess the usability process, none of them provides a self-assessment method considering the specific characteristics of…
Can Language Models Go Beyond Coding? Assessing the Capability of Language Models to Build Real-World Systems
Nankai University（南开大学）；Haihe Laboratory of Information Technology Application Innovation（海河信息技术应用创新实验室）；Key Laboratory of Data and Intelligent System Security, Ministry of Education（教育部数据与智能系统安全重点实验室）；Tianjin Key Laboratory of Software Experience and Human Computer Interaction（天津市软件体验与人机交互重点实验室）；Tsinghua University（清华大学）
Deep Learning (DL) models have been widely used to support code completion. These models, once properly trained, can take as input an incomplete code component (e.g., an incomplete function) and predict the missing tokens to finalize it. GitHub Copilot is an example of code recommender built by training a DL model on millions of open source repositories: The source code of these repositories acts as training data, allowing the model to learn "how to program". The usage of such a code is usually regulated by Free and Open Source Software (FOSS) licenses, that establish under which conditions th…
Modern programming languages (e.g., Java and C#) provide features to separate error-handling code from regular code, seeking to enhance software comprehensibility and maintainability. Nevertheless, the way exception handling (EH) code is structured in such languages may lead to multiple, different, and complex control flows, which may affect the software testability. Previous studies have reported that EH code is typically neglected, not well tested, and its misuse can lead to reliability degradation and catastrophic failures. However, little is known about the relationship between testing pra…
