CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology（哈尔滨工业大学社会计算与信息检索研究中心）；The School of Data and Computer Science, Sun Yat-sen University（中山大学数据与计算机科学学院）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Search Technology Center Asia（微软亚洲搜索技术中心）
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Ming Zhou
We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language codesearch, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing.
Improving Stack Overflow question title generation with copying enhanced CodeBERT model and bi-modal information
Wuhan University of Technology（武汉理工大学）；Wuhan University（武汉大学）；Wuhan University of Technology Chongqing Research Institute（武汉理工大学重庆研究院）；Sanya Science and Education Innovation Park of Wuhan University of Technology（武汉理工大学三亚科教创新园）；City University of Hong Kong（香港城市大学）
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Hefei University of Technology（合肥工业大学）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；University of Toronto（多伦多大学）
LangSelect: Cost-Aware Target-Language Routing for LLM Code Generation
RMIT University（皇家墨尔本理工大学）；Ho Chi Minh City University of Technology (HCMUT), VNU-HCM（越南国家大学胡志明市分校理工大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation
Wuhan University（武汉大学）；Noah’s Ark Lab, Huawei（华为诺亚方舟实验室）；Huazhong University of Science and Technology（华中科技大学）；Deakin University（迪肯大学）；Monash University（莫纳什大学）；Yunnan University（云南大学）
Better Language Models of Code through Self-Improvement
FPT Software AI Center（FPT软件人工智能中心）；Fulbright University（富布赖特大学）；McGill University（麦吉尔大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
