Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.
Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification
Southern University of Science and Technology(南方科技大学);The Chinese University of Hong Kong(香港中文大学);Xiamen University(厦门大学);City University of Hong Kong(香港城市大学)
Contrastive Knowledge Amalgamation for Unsupervised Image Classification
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Hangzhou Global Scientific and Technological Innovation Center(浙江大学杭州国际科创中心)
Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation
University of Technology Sydney(悉尼科技大学);Sun Yat-Sen University(中山大学);Peng Cheng National Lab(鹏城国家实验室);The University of Hong Kong(香港大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Multi-view Contrastive Learning for Online Knowledge Distillation
Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Xiamen Data Intelligence Academy of ICT, CAS(中国科学院计算技术研究所厦门数据智能研究院)
C2E: Boosting Ego-Only 3D Object Detection via Multi-Teacher Contrastive Knowledge Distillation
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University(福建省城市智能感知与计算重点实验室,厦门大学);Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(多媒体可信感知与高效计算教育部重点实验室,厦门大学);Zhongguancun Academy(中关村学院)
Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Zhongguancun Laboratory(中关村实验室);Horizon Robotics(地平线机器人)
Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training
Southeast University(东南大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Case Western Reserve University(凯斯西储大学);Centre de Recherche en Information Biomédicale Sino-Français(中法生物医学信息研究中心)
Enriching Knowledge Distillation with Intra-Class Contrastive Learning
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其跨学科应用重点实验室(东南大学),教育部,中国)
针对知识蒸馏中教师软标签缺乏类内多样表示的问题,本文提出带 margin loss 的类内对比学习教师训练方法,并通过理论与实验验证其有效性。
Contrastive Model Inversion for Data-Free Knowledge Distillation
Zhejiang University(浙江大学);Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies(浙江大学-阿里巴巴前沿技术联合研究中心);National University of Singapore(新加坡国立大学)
Preview-based Category Contrastive Learning for Knowledge Distillation
School of Computer Science and Technology, Shandong University(山东大学计算机科学与技术学院);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院);School of Software, Shandong University(山东大学软件学院);Alibaba Group(阿里巴巴集团)
Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);School of Data Science, University of Science and Technology of China(中国科学技术大学数据科学学院);iFLYTEK AI Research(讯飞AI研究院)
Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
The Chinese University of Hong Kong(香港中文大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Tencent AI Lab(腾讯AI实验室);University of Rochester(罗切斯特大学)
EVOKE: Elevating Chest X-ray Report Generation via Multi-View Contrastive Learning and Patient-Specific Knowledge
Xidian University(西安电子科技大学);Shanghai Jiao Tong University School of Medicine(上海交通大学医学院);Ruijin Hospital(瑞金医院);Shanghai Institute of Traumatology and Orthopedics(上海市伤骨科研究所);Harbin Medical University(哈尔滨医科大学);The First Affiliated Hospital of Harbin Medical University(哈尔滨医科大学附属第一医院);Brown University(布朗大学)
Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution
Korea Electronics Technology Institutes(韩国电子技术研究院)
最近,基于CNN的SISR具有许多参数和高计算成本以实现更好的性能,限制了其对资源受限设备(如移动设备)的适用性。知识蒸馏(KD)是使网络高效的方法之一,它将教师的有用知识传递给学生,目前正在研究中。最近,KD for SISR利用特征蒸馏(FD)来最小化教师和学生网络之间特征图的欧氏距离损失,但它没有充分考虑如何有效地和有意义地从教师那里传递知识,以提高学生在给定网络容量限制下的表现。在本文中,我们提出了一个特征域自适应对比蒸馏(FACD)方法,有效地训练轻量级学生SISR网络。我们使用欧氏距离损失现有FD方法的局限性,并提出了一个特征域对比损失,使学生网络学习更丰富的信息,从教师的代表在特征域。此外,我们提出了一种自适应蒸馏,根据训练补丁的条件选择性地应用蒸馏。实验结果表明,学生EDSR和RCAN网络与建议的FACD计划不仅提高了整个基准数据集和规模的PSNR性能,而且与传统的FD方法相比,主观图像质量。
Low-Resolution Object Recognition with Cross-Resolution Relational Contrastive Distillation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Baidu Inc.(百度公司);School of Cyber Security at University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Department of Communication Engineering, Shanghai University(上海大学通信工程系)
DiffAug: Enhance Unsupervised Contrastive Learning with Domain-Knowledge-Free Diffusion-based Data Augmentation
Westlake University(西湖大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);National University of Singapore(新加坡国立大学);Hupan Lab, Zhejiang Province(浙江省湖畔实验室);AI Lab, Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心人工智能实验室)