Training on large-scale graphs has achieved remarkable results in graph representation learning, but its cost and storage have attracted increasing concerns. Existing graph condensation methods primarily focus on optimizing the feature matrices of condensed graphs while overlooking the impact of the structure information from the original graphs. To investigate the impact of the structure information, we conduct analysis from the spectral domain and empirically identify substantial Laplacian Energy Distribution (LED) shifts in previous works. Such shifts lead to poor performance in cross-architecture generalization and specific tasks, including anomaly detection and link prediction. In this paper, we propose a novel Structure-broadcasting Graph Dataset Distillation (SGDD) scheme for broadcasting the original structure information to the generation of the synthetic one, which explicitly prevents overlooking the original structure information. Theoretically, the synthetic graphs by SGDD are expected to have smaller LED shifts than previous works, leading to superior performance in both cross-architecture settings and specific tasks. We validate the proposed SGDD across 9 datasets and achieve state-of-the-art results on all of them: for example, on the YelpChi dataset, our approach maintains 98.6% test accuracy of training on the original graph dataset with 1,000 times saving on the scale of the graph. Moreover, we empirically evaluate there exist 17.6% ~ 31.4% reductions in LED shift crossing 9 datasets. Extensive experiments and analysis verify the effectiveness and necessity of the proposed designs. The code is available in the GitHub repository: https://github.com/RingBDStack/SGDD.
DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Kuaishou Technology(快手科技);Zhongnan University of Economics and Law(中南财经政法大学)
Seeing in the Dark: A Teacher-Student Framework for Dark Video Action Recognition via Knowledge Distillation and Contrastive Learning
School of Information Technology, Monash University, Malaysia;Faculty of Information Technology, Monash University, Australia;Robotics \& Autonomous Systems Lab, TCS Research, India;Dept of Electrical Engineering, University of Malaya, Malaysia
Path-Guided Flow Matching for Dataset Distillation
Department of Machine Learning, MBZUAI, Abu Dhabi, UAE(机器学习系,MBZUAI,阿布扎赫德,阿联酋);Department of Computer Vision, MBZUAI, Abu Dhabi, UAE(计算机视觉系,MBZUAI,阿布扎赫德,阿联酋)
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
Google Research(谷歌研究院);Nvidia(英伟达)
我们的目标是推进开放词汇目标检测,它检测任意文本输入描述的对象。根本的挑战是能否获得训练数据。进一步扩大现有目标检测数据集中包含的类的数量是昂贵的。为了克服这一挑战,我们提出了ViLD,一种通过视觉和语言知识蒸馏的训练方法。我们的方法将知识从预先训练的开放词汇图像分类模型(教师)中提取到两阶段检测器(学生)中。具体来说,我们使用教师模型来编码类别文本和对象提案的图像区域。然后,我们训练一个学生检测器,其检测到的盒子的区域嵌入与教师推断的文本和图像嵌入对齐。我们在LVIS上进行基准测试,将所有罕见类别作为训练过程中看不到的新类别。ViLD使用ResNet-50骨干网获得16.1掩码AP $_r$,甚至比有监督的对手高出3.8。当使用更强的教师模型ALIGN训练时,ViLD达到26.3 AP $_r$。该模型可以直接移植到其他数据集,无需微调,在PASCAL VOC上达到72.2 AP $_{50}$,在COCO上达到36.6 AP,在Objects 365上达到11.8 AP。在COCO上,ViLD在新型AP上的表现优于先前的最先进水平4.8,在整体AP上优于先前的最先进水平11.4。代码和演示在https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild上开源。
MoPD: Mixture-of-Prompts Distillation for Vision-Language Models
Department of Computer Science and Engineering, Southern University of Science and Technology(计算机科学与工程系,南方科技大学);School of Mathematical and Computer Sciences, University of Adelaide(数学与计算机科学学院,阿德莱德大学)