本文提出Order-Expressive Masked Diffusion Model (OeMDM)和Learnable-Order Masked Diffusion Model (LoMDM)，统一了不同生成顺序的扩散生成过程，并通过单目标学习生成顺序和扩散骨干，提升了文本生成性能。
摘要：
Masked diffusion models (MDMs) 是语言生成中替代自回归模型 (ARMs) 的潜在选择，但生成质量严重依赖于生成顺序。先前工作要么硬编码顺序（例如块状左到右），要么为预训练的MDM学习顺序策略，这会带来额外成本并可能导致次优解，因为存在两阶段优化。受此启发，我们提出了order-expressive masked diffusion model (OeMDM)，以适用于各种生成顺序的广泛扩散生成过程，使MDM、ARM和块扩散能在单一框架中进行解释。此外，基于OeMDM，我们引入了learnable-order masked diffusion model (LoMDM)，通过单目标学习生成顺序和扩散骨干，使扩散模型能够根据上下文生成顺序进行文本生成。实证上，我们证实LoMDM在多个语言模型基准测试中优于各种离散扩散模型。
英文摘要：
Masked diffusion models (MDMs) are a potential alternative to autoregressive models (ARMs) for language generation, but generation quality depends critically on the generation order. Prior work either hard-codes an ordering (e.g., blockwise left-to-right) or learns an ordering policy for a pretrained MDM, which incurs extra cost and can yield suboptimal solutions due to the two-stage optimization. Motivated by this, we propose order-expressive masked diffusion model (OeMDM) for a broad class of diffusion generative processes with various generation orders, enabling the interpretation of MDM, ARM, and block diffusion in a single framework. Furthermore, building on OeMDM, we introduce learnable-order masked diffusion model (LoMDM), which jointly learns the generation ordering and diffusion backbone through a single objective from scratch, enabling the diffusion model to generate text in context-dependent ordering. Empirically, we confirm that LoMDM outperforms various discrete diffusion models across multiple language modeling benchmarks.
Department of Computer Science, Cornell Tech, NYC, USA（康奈尔科技学院计算机科学系）；Department of Computer Science, Cornell University, Ithaca, USA（康奈尔大学计算机科学系）；School of Computer and Communication Sciences, EPFL Lausanne, Switzerland（洛桑联邦理工学院计算机与通信科学学院）；Department of Computer Science, University of Chicago, Illinois（芝加哥大学计算机科学系）；NVIDIA, Santa Clara, USA（英伟达）
National University of Singapore（新加坡国立大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper presents a novel nonmyopic adaptive Gaussian process planning (GPP) framework endowed with a general class of Lipschitz continuous reward functions that can unify some active learning/sensing and Bayesian optimization criteria and offer practitioners some flexibility to specify their desired choices for defining new tasks/problems. In particular, it utilizes a principled Bayesian sequential decision problem framework for jointly and naturally optimizing the exploration-exploitation trade-off. In general, the resulting induced GPP policy cannot be derived exactly due to an uncountabl…
Mila（米拉人工智能研究所）；University of Montreal（蒙特利尔大学）；Polytechnique Montreal（蒙特利尔理工学院）；ServiceNow Research（ServiceNow研究院）
Video prediction is a challenging task. The quality of video frames from current state-of-the-art (SOTA) generative models tends to be poor and generalization beyond the training data is difficult. Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a general-purpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using a probabilistic conditional score-based denoising diffusion model, conditioned on …
Masked autoencoders are scalable vision learners, as the title of MAE \cite{he2022masked}, which suggests that self-supervised learning (SSL) in vision might undertake a similar trajectory as in NLP. Specifically, generative pretext tasks with the masked prediction (e.g., BERT) have become a de facto standard SSL practice in NLP. By contrast, early attempts at generative methods in vision have been buried by their discriminative counterparts (like contrastive learning); however, the success of mask image modeling has revived the masking autoencoder (often termed denoising autoencoder in the pa…
Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Facebook AI Research (Meta AI)（脸书人工智能研究院（元人工智能））；CIFAR（加拿大高等研究院）
There are many frameworks for deep generative modeling, each often presented with their own specific training algorithms and inference methods. Here, we demonstrate the connections between existing deep generative models and the recently introduced GFlowNet framework, a probabilistic inference machine which treats sampling as a decision-making process. This analysis sheds light on their overlapping traits and provides a unifying viewpoint through the lens of learning with Markovian trajectories. Our framework provides a means for unifying training and inference algorithms, and provides a route…
Mila；University of Montreal（蒙特利尔大学）；CIFAR Fellow（CIFAR研究员）
Black-box optimization formulations for biological sequence design have drawn recent attention due to their promising potential impact on the pharmaceutical industry. In this work, we propose to unify two seemingly distinct worlds: likelihood-free inference and black-box optimization, under one probabilistic framework. In tandem, we provide a recipe for constructing various sequence design methods based on this framework. We show how previous optimization approaches can be "reinvented" in our framework, and further propose new probabilistic black-box optimization algorithms. Extensive experime…
Microsoft（微软）；University of Washington（华盛顿大学）；Meta AI；UCLA（加州大学洛杉矶分校）
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but …
Beyond Labels: A Self-Supervised Framework with Masked Autoencoders and Random Cropping for Breast Cancer Subtype Classification
DISIT, Computer Science Institute, University of Piemonte Orientale（DISIT计算机科学研究所，皮埃蒙特东方大学）；Center for Translational Research on Autoimmune and Allergic Diseases (CAAD), Department of Health Sciences, University of Piemonte Orientale（自身免疫和过敏疾病转化研究中心（CAAD），健康科学系，皮埃蒙特东方大学）；Interdisciplinary Research Center of Autoimmune Diseases (IRCAD), Department of Health Sciences, University of Piemonte Orientale（自身免疫疾病跨学科研究中心（IRCAD），健康科学系，皮埃蒙特东方大学）
提出一种基于 MAE 与随机裁剪的自监督框架，从 WSI 学习组织病理图像表示，并通过线性探针完成乳腺癌多亚型分类。
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）
Coordinate-MLPs are emerging as an effective tool for modeling multidimensional continuous signals, overcoming many drawbacks associated with discrete grid-based approximations. However, coordinate-MLPs with ReLU activations, in their rudimentary form, demonstrate poor performance in representing signals with high fidelity, promoting the need for positional embedding layers. Recently, Sitzmann et al. proposed a sinusoidal activation function that has the capacity to omit positional embedding from coordinate-MLPs while still preserving high signal fidelity. Despite its potential, ReLUs are stil…
MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZIU-UIUC Institute, Zhejiang University（浙江大学ZIU-UIUC研究院）；School of Public Health, Zhejiang University（浙江大学公共卫生学院）
