SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation
Wuhan University（武汉大学）；Noah’s Ark Lab, Huawei（华为诺亚方舟实验室）；Huazhong University of Science and Technology（华中科技大学）；Deakin University（迪肯大学）；Monash University（莫纳什大学）；Yunnan University（云南大学）
Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, Xin Jiang
Code representation learning, which aims to encode the semantics of source code into distributed vectors, plays an important role in recent deep-learning-based models for code intelligence. Recently, many pre-trained language models for source code (e.g., CuBERT and CodeBERT) have been proposed to model the context of code and serve as a basis for downstream code intelligence tasks such as code search, code clone detection, and program translation. Current approaches typically consider the source code as a plain sequence of tokens, or inject the structure information (e.g., AST and data-flow) into the sequential model pre-training. To further explore the properties of programming languages, this paper proposes SynCoBERT, a syntax-guided multi-modal contrastive pre-training approach for better code representations. Specially, we design two novel pre-training objectives originating from the symbolic and syntactic properties of source code, i.e., Identifier Prediction (IP) and AST Edge Prediction (TEP), which are designed to predict identifiers, and edges between two nodes of AST, respectively. Meanwhile, to exploit the complementary information in semantically equivalent modalities (i.e., code, comment, AST) of the code, we propose a multi-modal contrastive learning strategy to maximize the mutual information among different modalities. Extensive experiments on four downstream tasks related to code intelligence show that SynCoBERT advances the state-of-the-art with the same pre-training corpus and model size.
Wuhan University（武汉大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Huazhong University of Science and Technology（华中科技大学）；Monash University（莫纳什大学）；Yunnan University（云南大学）
Recent years have witnessed increasing interest in code representation learning, which aims to represent the semantics of source code into distributed vectors. Currently, various works have been proposed to represent the complex semantics of source code from different views, including plain text, Abstract Syntax Tree (AST), and several kinds of code graphs (e.g., Control/Data Flow Graph). However, most of them only consider a single view of source code independently, ignoring the correspondences among different views. In this paper, we propose to integrate different views with the natural-lang…
Improving Code Translation with Syntax-Guided and Semantic-aware Preference Optimization
State Key Laboratory for Novel Software Technology, Nanjing University, China（南京大学新型软件技术国家重点实验室）；National Institute of Healthcare Data Science, Nanjing University, China（南京大学健康数据科学国家研究院）
Modelbest；Tsinghua University（清华大学）；University College London（伦敦大学学院）
Inductive knowledge graph completion requires models to comprehend the underlying semantics and logic patterns of relations. With the advance of pretrained language models, recent research have designed transformers for link prediction tasks. However, empirical studies show that linearizing triples affects the learning of relational patterns, such as inversion and symmetry. In this paper, we propose Bi-Link, a contrastive learning framework with probabilistic syntax prompts for link predictions. Using grammatical knowledge of BERT, we efficiently search for relational prompts according to lear…
SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRI
College of Electronic Information and Optical Engineering, Nankai University（南开大学电子信息与光学工程学院）；Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology（天津市光电传感器与传感网络技术重点实验室）；Tianjin First Central Hospital, School of Medicine, Nankai University（南开大学医学院附属天津市第一中心医院）
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）
Change captioning is to describe the semantic change between a pair of similar images in natural language. It is more challenging than general image captioning, because it requires capturing fine-grained change information while being immune to irrelevant viewpoint changes, and solving syntax ambiguity in change descriptions. In this paper, we propose a neighborhood contrastive transformer to improve the model's perceiving ability for various changes under different scenes and cognition ability for complex syntax structure. Concretely, we first design a neighboring feature aggregating to integ…
TransformCode: A Contrastive Learning Framework for Code Embedding via Subtree Transformation
School of Computer Science and Engineering, Macau University of Science and Technology（澳门科技大学计算机科学与工程学院）；School of Computer Science, University of Nottingham Ningbo China（宁波诺丁汉大学计算机学院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代软件技术国家重点实验室）
Fudan University（复旦大学）；Microsoft（微软）；National University of Singapore（新加坡国立大学）
Code contrastive pre-training has recently achieved significant progress on code-related tasks. In this paper, we present \textbf{SCodeR}, a \textbf{S}oft-labeled contrastive pre-training framework with two positive sample construction methods to learn functional-level \textbf{Code} \textbf{R}epresentation. Considering the relevance between codes in a large-scale code corpus, the soft-labeled contrastive pre-training can obtain fine-grained soft-labels through an iterative adversarial manner and use them to learn better code representation. The positive sample construction is another key for c…
Unveiling Large Language Models Generated Texts: A Multi-Level Fine-Grained Detection Framework
School of Information, Renmin University of China（中国人民大学信息学院）；Institute for Advanced Algorithms Research, Shanghai（上海高级算法研究院）；School of Information Technology and Management, University of International Business and Economics（对外经济贸易大学信息技术与管理学院）；Shanghai University of Finance and Economics（上海财经大学）
Domain-Adaptive Pre-Training for Arabic Aspect-Based Sentiment Analysis: A Comparative Study of Domain Adaptation and Fine-Tuning Strategies
Department of Computer Science, Faculty of Computing and Information Technology, King Abdulaziza University, Jeddah, Saudi Arabia（计算机科学系，计算与信息技术学院，国王阿卜杜勒阿齐兹大学，沙特阿拉伯吉达）
