在推理时间算法中,预训练模型被用作子程序来解决下游任务,已成为一种新兴范式。此类算法已被提出用于从逆问题、引导图像生成到推理等任务。然而,目前实际部署的方法都是启发式方法,具有各种失败模式——我们对这些启发式方法何时可以高效改进的理解非常有限。在本文中,我们考虑从奖励倾斜的扩散模型中采样——即从p*(x) ∝ p(x) exp(r(x))中采样——给定一个奖励函数r和预训练的扩散 oracle 用于p。我们对二次奖励r(x) = x^T A x + b^T x的任务的计算可处理性 进行细粒度分析。我们证明线性奖励倾斜总是可以高效采样——这是一个简单的结果,似乎在文献中未被注意到。我们使用这个作为基础块,结合一个概念上新的成分——Hubbard-Stratonovich变换——来提供一种高效算法,用于从低秩正定二次倾斜中采样,即r(x) = x^T A x,其中A是正定且秩为O(1)。对于负定倾斜,即r(x) = -x^T A x,其中A是正定的,我们证明即使A的秩为1(尽管具有指数级大的条目),该问题也是不可行的。
英文摘要:
Inference-time algorithms are an emerging paradigm in which pre-trained models are used as subroutines to solve downstream tasks. Such algorithms have been proposed for tasks ranging from inverse problems and guided image generation to reasoning. However, the methods currently deployed in practice are heuristics with a variety of failure modes -- and we have very little understanding of when these heuristics can be efficiently improved.
In this paper, we consider the task of sampling from a reward-tilted diffusion model -- that is, sampling from $p^{\star}(x) \propto p(x) \exp(r(x))$ -- given a reward function $r$ and pre-trained diffusion oracle for $p$. We provide a fine-grained analysis of the computational tractability of this task for quadratic rewards $r(x) = x^\top A x + b^\top x$. We show that linear-reward tilts are always efficiently sampleable -- a simple result that seems to have gone unnoticed in the literature. We use this as a building block, along with a conceptually new ingredient -- the Hubbard-Stratonovich transform -- to provide an efficient algorithm for sampling from low-rank positive-definite quadratic tilts, i.e. $r(x) = x^\top A x$ where $A$ is positive-definite and of rank $O(1)$. For negative-definite tilts, i.e. $r(x) = - x^\top A x$ where $A$ is positive-definite, we prove that the problem is intractable even if $A$ is of rank 1 (albeit with exponentially-large entries).
Euphonium: Steering Video Flow Matching via Process Reward Gradient Guided Stochastic Dynamics
Huazhong University of Science and Technology(华中科技大学);Shanghai Jiao Tong University(上海交通大学);Tencent Hunyuan(腾讯混元);University of Chinese Academy of Sciences(中国科学院大学)
Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction
Université de Montréal(蒙特利尔大学);Mila(米拉计算科学研究所);Dreamfold(Dreamfold公司);Duke University(杜克大学);McGill University(麦吉尔大学);Allen Institute for AI(艾伦人工智能研究所);Oxford University(牛津大学);Aithyra(Aithyra公司)
Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)科学与工程学院);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)
PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Soochow University(苏州大学);Stony Brook University(石溪大学);The Chinese University of Hong Kong(香港中文大学)
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
University of Cambridge(剑桥大学);Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院);Field AI;University of Washington(华盛顿大学);University of Tennessee(田纳西大学);Korean Advanced Institute of Science and Technology(韩国高等科学技术院);University of California San Diego(加州大学圣地亚哥分校);The Alan Turing Institute(艾伦·图灵研究所);ServiceNow