The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling
The Hong Kong University of Science and Technology(香港科技大学);University of Chinese Academy of Sciences(中国科学院大学);Academy of Mathematics and Systems Science(数学与系统科学研究院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);University of Electronic Science and Technology of China(电子科技大学);National University of Singapore(新加坡国立大学)
With the incorporation of the UNet architecture, diffusion probabilistic models have become a dominant force in image generation tasks. One key design in UNet is the skip connections between the encoder and decoder blocks. Although skip connections have been shown to improve training stability and model performance, we reveal that such shortcuts can be a limiting factor for the complexity of the transformation. As the sampling steps decrease, the generation process and the role of the UNet get closer to the push-forward transformations from Gaussian distribution to the target, posing a challenge for the network's complexity. To address this challenge, we propose Skip-Tuning, a simple yet surprisingly effective training-free tuning method on the skip connections. Our method can achieve 100% FID improvement for pretrained EDM on ImageNet 64 with only 19 NFEs (1.75), breaking the limit of ODE samplers regardless of sampling steps. Surprisingly, the improvement persists when we increase the number of sampling steps and can even surpass the best result from EDM-2 (1.58) with only 39 NFEs (1.57). Comprehensive exploratory experiments are conducted to shed light on the surprising effectiveness. We observe that while Skip-Tuning increases the score-matching losses in the pixel space, the losses in the feature space are reduced, particularly at intermediate noise levels, which coincide with the most effective range accounting for image quality improvement.
Dynamic Network Embedding via Incremental Skip-gram with Negative Sampling
Beihang University(北京航空航天大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);National Computer Network Emergency Response Technical Team/Coordination Center of China(国家计算机网络应急技术处理协调中心);University of Southern California(南加州大学)
Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets
Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);PCL Research Center of Networks and Communications, Peng Cheng Laboratory(鹏城实验室网络通信PCL研究中心);The University of Melbourne(墨尔本大学)
Bypassing Skip-Gram Negative Sampling: Dimension Regularization as a More Efficient Alternative for Graph Embeddings
Northeastern University(东北大学);Khoury College of Computer Sciences(计算机科学学院);Amazon AWS(亚马逊AWS);San Francisco CA USA(旧金山CA美国);Network Science Institute(网络科学研究所);Stanford University(斯坦福大学);Management Science & Engineering(管理科学与工程);Stanford CA USA(斯坦福CA美国)
本文提出用维度正则化替代 Skip-Gram 负采样来防止图嵌入坍缩,并构建算法增强框架以提升 LINE 和 node2vec 的可扩展性,在保持链路预测性能的同时显著降低 GPU 内存占用和训练时间。
FLEX: A Backbone for Diffusion-Based Modeling of Spatio-temporal Physical Systems
Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);International Computer Science Institute(国际计算机科学研究所);University of California, Berkeley(加州大学伯克利分校);Ben-Gurion University of the Negev(贝内-约瑟夫大学);KTH Royal Institute of Technology(皇家理工学院)