代码大语言模型(Code Large Language Models,Code LLMs)在适应代码生成与补全、缺陷检测与修复等各类下游任务方面表现出很强的通用性。然而,Code LLMs 往往无法把握既有编码规范,导致生成的代码与给定项目所要求的设计模式相冲突。因此,开发者必须进行后处理,使生成代码符合项目的设计规范。在本研究中,我们通过实证方式考察 Code LLMs 在软件开发中的偏差。通过精心设计的实验,我们从识别、理解和生成三个方面评估模型对设计模式的理解。研究结果表明,Code LLMs 中的偏差会显著影响下游任务的可靠性。
英文摘要:
Code Large Language Models (LLMs) demonstrate great versatility in adapting to various downstream tasks, including code generation and completion, as well as bug detection and fixing. However, Code LLMs often fail to capture existing coding standards, leading to the generation of code that conflicts with the required design patterns for a given project. As a result, developers must post-process to adapt the generated code to the project's design norms. In this work, we empirically investigate the biases of Code LLMs in software development. Through carefully designed experiments, we assess the models' understanding of design patterns across recognition, comprehension, and generation. Our findings reveal that biases in Code LLMs significantly affect the reliability of downstream tasks.
Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs
Saudi Data & AI Authority (SDAIA)(沙特数据与人工智能局);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学);Princess Nourah Bint Abdulrahman University (PNU)(普罗瑟·努拉·本·阿卜杜勒拉赫曼大学)
Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment
Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Beijing Institute for General Artificial Intelligence(北京一般人工智能研究所)
The Dark Patterns of Personalized Persuasion in Large Language Models: Exposing Persuasive Linguistic Features for Big Five Personality Traits in LLMs Responses
Warsaw University of Technology(华沙理工大学);University of Warsaw(华沙大学)
Pun Unintended: LLMs and the Illusion of Humor Understanding
Dept of Environmental Sciences, Informatics and Statistics, Ca’ Foscari University of Venice(环境科学、信息与统计系,威尼斯卡弗西大学);School of Computer Science and Informatics, Cardiff University(计算机科学与信息学院,卡迪夫大学)
CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
Department of Informatics(信息学院);King’s College London(伦敦国王学院);Institute of Psychology(心理学研究所);University of Tartu(塔尔图大学);Department of Psychology(心理学系);Stanford University(斯坦福大学);The Alan Turing Institute(艾伦·图灵研究所)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMs
University of Cambridge(剑桥大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Mila, University of Montreal(蒙特利尔大学Mila研究所);Google DeepMind(谷歌DeepMind)
How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs
The University of Osaka(大阪大学);UC Davis(加州大学戴维斯分校);UC Berkeley(加州大学伯克利分校);Peking University(北京大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Shanghai Jiao Tong University(上海交通大学);NII(信息学研究所(日本));City University of Hong Kong(香港城市大学)
Patterns vs. Patients: Evaluating LLMs against Mental Health Professionals on Personality Disorder Diagnosis through First-Person Narratives
IDEAS Research Institute(IDEAS研究 institute);Adam Mickiewicz University(亚当·密茨凯维奇大学);AMU Center for Artificial Intelligence(AMU人工智能中心);Poznań University of Medical Sciences(波兹南医学科学大学);Maria Curie-Skłodowska University(玛丽·居里-斯克洛多夫斯卡大学)
Rethinking the Understanding Ability across LLMs through Mutual Information
Department of Industrial and Systems Engineering(工业与系统工程系);University of Houston(休斯敦大学);Bakar Computational Health Sciences Institute(巴卡计算健康科学研究所);UCSF
SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Lab of Knowledge Graph(浙江大学蚂蚁集团知识图谱联合实验室);Ant Group(蚂蚁集团)
SciTS: Scientific Time Series Understanding and Generation with LLMs
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学)