一种使用软掩码噪声的更廉价且更好的扩散语言模型
A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
AI总结:
针对现有扩散模型处理离散语言数据的局限性,提出 Masked-Diffuse LM,通过语言学感知的软掩码前向过程和交叉熵损失直接预测类别分布,以更低训练成本和更高效率实现了更优的生成质量。
摘要:
基于迭代去噪的扩散模型近期被提出并应用于图像生成等多种生成任务中。然而,作为本质上为连续数据构建的方法,现有的扩散模型在建模离散数据(例如语言)时仍存在一些局限性。例如,常用的高斯噪声无法很好地处理离散破坏,且连续空间中的目标函数在扩散过程中对文本数据难以保持稳定,尤其是在维度较高时。为了缓解这些问题,受语言中的语言学特征启发,我们引入了一种用于语言建模的新型扩散模型 Masked-Diffuse LM,其具有更低的训练成本和更好的性能。具体而言,我们设计了一个语言学感知的前向过程,通过策略性地软掩码(soft-masking)向文本添加破坏,以更好地对文本数据进行加噪。同时,我们在每个扩散步骤中直接使用交叉熵损失函数预测类别分布,以更高效、直接的方式连接连续空间和离散空间。通过在 5 个受控生成任务上的实验,我们证明了我们的 Masked-Diffuse LM 能够以更高的效率实现比最先进扩散模型更好的生成质量。
英文摘要:
Diffusion models that are based on iterative denoising have been recently proposed and leveraged in various generation tasks like image generation. Whereas, as a way inherently built for continuous data, existing diffusion models still have some limitations in modeling discrete data, e.g., languages. For example, the generally used Gaussian noise can not handle the discrete corruption well, and the objectives in continuous spaces fail to be stable for textual data in the diffusion process especially when the dimension is high. To alleviate these issues, we introduce a novel diffusion model for language modeling, Masked-Diffuse LM, with lower training cost and better performances, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. Also, we directly predict the categorical distribution with cross-entropy loss function in every diffusion step to connect the continuous space and discrete space in a more efficient and straightforward way. Through experiments on 5 controlled generation tasks, we demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。