Current auto-regressive (AR) LLMs, diffusion-based text/image generative models, and recent flow matching (FM) algorithms are capable of generating premium quality text/image samples. However, the inference or sample generation in these models is often very time-consuming and computationally demanding, mainly due to large numbers of function evaluations corresponding to the lengths of tokens or the numbers of diffusion steps. This also necessitates heavy GPU resources, time, and electricity. In this work we propose a novel solution to reduce the sample generation time of flow matching algorithms by a guaranteed speed-up factor, without sacrificing the quality of the generated samples. Our key idea is to utilize computationally lightweight generative models whose generation time is negligible compared to that of the target AR/FM models. The draft samples from a lightweight model, whose quality is not satisfactory but fast to generate, are regarded as an initial distribution for a FM algorithm. Unlike conventional usage of FM that takes a pure noise (e.g., Gaussian or uniform) initial distribution, the draft samples are already of decent quality, so we can set the starting time to be closer to the end time rather than 0 in the pure noise FM case. This will significantly reduce the number of time steps to reach the target data distribution, and the speed-up factor is guaranteed. Our idea, dubbed {\em Warm-Start FM} or WS-FM, can essentially be seen as a {\em learning-to-refine} generative model from low-quality draft samples to high-quality samples. As a proof of concept, we demonstrate the idea on some synthetic toy data as well as real-world text and image generation tasks, illustrating that our idea offers guaranteed speed-up in sample generation without sacrificing the quality of the generated samples.
Learning to Solve the Quadratic Assignment Problem with Warm-Started MCMC Finetuning
School of Mathematical Science, Peking University(北京大学数学科学学院);State Key Laboratory of Mathematical Sciences, Institute of Computational Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院数学科学国家重点实验室);Beijing International Center for Mathematical Research, Peking University(北京大学北京国际数学研究中心)
Rethinking Warm-Starts with Predictions: Learning Predictions Close to Sets of Optimal Solutions for Faster $\text{L}$-/$\text{L}^\natural$-Convex Function Minimization
NOWS: Neural Operator Warm Starts for Accelerating Iterative Solvers
Leibniz University Hannover(莱布尼茨汉诺威大学);Institute of Photonics(光子研究所);Department of Mathematics and Physics(数学与物理系);Tsinghua University(清华大学);Institute of Structural Mechanics(结构力学研究所);Bauhaus-Universität Weimar(魏玛建筑学院)
Fast, Scalable, Warm-Start Semidefinite Programming with Spectral Bundling and Sketching
Manning College of Information and Computer Sciences, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校信息与计算机科学学院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)
Short-term Load Forecasting Based on Hybrid Strategy Using Warm-start Gradient Tree Boosting
Southeast University(东南大学);National University of Singapore(新加坡国立大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
提出了一种基于深度学习的短期负荷预测混合策略。该策略提出了一种新的基于树的集成方法Warm-start Gradient Tree Boosting(WGTB)。目前的策略要么是单一类型的集成子模型,无法利用不同推理模型的统计优势。或者他们只是简单地将完全不同的推理模型的输出相加,这并不能最大限度地发挥集成的潜力。受偏差-方差权衡的启发,WGTB被提出,并针对不同推理模型在准确性、波动性和线性度上的巨大差异而定制。完整的策略整合了四种不同能力的推理模型。然后,WGTB通过热启动以及装袋和提升的混合来集成它们的输出,从而同时降低偏差和方差。在国家电网公司的两个小时分辨率的实际数据集上进行了验证。结果表明,所提出的策略的有效性,杂交的低偏差和低方差推理模型的统计优势。
Warm Up Cold-start Advertisements: Improving CTR Predictions via Learning to Learn ID Embeddings
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);IIIS, Tsinghua University(清华大学交叉信息研究院)
Variational quantum simulation: a case study for understanding warm starts
Institute of Physics, Ecole Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院物理研究所);IBM Quantum, IBM Research – Zurich(IBM量子(IBM苏黎世研究院));Chulalongkorn University(朱拉隆功大学)
AppQ: Warm-starting App Recommendation Based on View Graphs
Beijing Jiaotong University(北京交通大学);The Pennsylvania State University(宾夕法尼亚州立大学);King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学)
Deconfounded Warm-Start Thompson Sampling with Applications to Precision Medicine
Daniels School of Business, Purdue University(普渡大学 Daniels 商学院);Mays Business School, Texas A&M University(德克萨斯农工大学 Mays 商学院);McCombs School of Business, University of Texas at Austin(德克萨斯大学奥斯汀分校 McCombs 商学院)
LAWS: Look Around and Warm-Start Natural Gradient Descent for Quantum Neural Networks
William & Mary(威廉与玛丽学院)
变分量子算法(VQA)最近受到了研究界的极大关注,因为它们在有噪声的中间尺度量子计算机(NISQ)中有很好的性能。然而,在具有随机初始化参数的参数化量子线路(PQC)上运行的VQA的特征在于贫瘠高原(BP),其中梯度在量子比特的数量上呈指数地消失。在本文中,我们首先回顾量子自然梯度(QNG),这是最流行的算法之一,在VQA,从经典的一阶优化的角度来看。然后,我们提出了一个\underline{L} ook \underline{A} round \underline{W} arm- \underline{S} tart QNG(LAWS)算法,以减轻广泛存在的BP问题。LAWS是一种组合优化策略,它利用了模型参数初始化和QNG快速收敛的优点。LAWS重复地重新调整参数搜索空间,以用于下一次迭代参数更新。重新初始化的参数搜索空间是通过对接近当前最优的梯度进行采样来仔细选择的。此外,我们提出了一个统一的框架(WS-SGD)集成参数初始化技术到优化器。我们提供了收敛性证明的建议框架的凸和非凸目标函数的基础上Polyak-Lojasiewicz(PL)条件。实验结果表明,该算法在量子分类问题中能有效地降低BP算法的泛化能力。