arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.15595cs.LGcs.AI
收藏

离散流匹配

Discrete Flow Matching

Meta AI;FAIR;Weizmann Institute(魏茨曼研究所)

Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, Yaron Lipman

AI总结:

离散流匹配提出一种新型离散流生成范式,通过通用概率路径与学习后验采样,在1.7B参数下于HumanEval和MBPP上取得显著性能,以非自回归方式缩小与自回归模型的差距。

摘要:

尽管流匹配和扩散模型已成为图像和视频等连续变量的强大生成范式,但它们在高维离散数据(如语言)上的应用仍然有限。在这项工作中,我们提出了离散流匹配(Discrete Flow Matching),一种专门为生成离散数据而设计的新型离散流范式。离散流匹配提供了几个关键贡献:(i)它适用于在源分布和目标分布之间插值的一般概率路径族;(ii)它允许使用学习到的后验(如概率去噪器(x-预测)和噪声预测(ε-预测))从这些概率路径采样的通用公式;(iii)在实践中,关注由不同调度器定义的特定概率路径,与先前的离散扩散和流模型相比,改善了生成困惑度;(iv)通过将离散流匹配模型扩展到1.7B参数,我们在HumanEval上达到6.7%的Pass@1和13.4%的Pass@10,在1-shot MBPP编码基准上达到6.7%的Pass@1和20.6%的Pass@10。我们的方法能够以非自回归方式生成高质量的离散数据,显著缩小了自回归模型与离散流模型之间的差距。

英文摘要:

Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited. In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data. Discrete Flow Matching offers several key contributions:(i) it works with a general family of probability paths interpolating between source and target distributions; (ii) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the probability denoiser ($x$-prediction) and noise-prediction ($ε$-prediction); (iii) practically, focusing on specific probability paths defined with different schedulers improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) by scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autoregressive fashion, significantly closing the gap between autoregressive models and discrete flow models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑