扩散语言模型可以通过缩放和指令微调执行许多任务
Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
AI总结：
本文证明通过数据、规模和任务上的扩展，扩散语言模型能成为强大的语言学习者，并通过指令微调激发上下文学习能力，解决多种一般语言任务。
摘要：
近年来生成式人工智能的迅猛发展得益于扩散概率模型的生成能力和大型语言模型的可扩展能力。尽管潜力巨大，扩散语言模型能否解决与自回归模型相当的一般语言任务仍不明确。本文证明，在数据、规模和任务方面扩展扩散模型可以有效使其成为强大的语言学习者。我们首先通过掩码语言建模预训练从海量数据中获取知识，利用它们之间的内在联系，构建了规模上具有竞争力的扩散语言模型。然后，我们通过扩散适配将预训练的掩码语言模型重新编程为扩散语言模型，其中探索了任务特定微调和指令微调，以解锁它们在解决一般语言任务方面的多功能性。实验表明，扩展扩散语言模型持续提高了下游语言任务的性能。我们进一步发现，指令微调可以激发零样本和少样本的上下文学习能力，有助于通过遵循自然语言指令处理许多未见过的任务，并在推理等高级和具有挑战性的能力方面展现出前景。
英文摘要：
The recent surge of generative AI has been fueled by the generative power of diffusion probabilistic models and the scalable capabilities of large language models. Despite their potential, it remains elusive whether diffusion language models can solve general language tasks comparable to their autoregressive counterparts. This paper demonstrates that scaling diffusion models w.r.t. data, sizes, and tasks can effectively make them strong language learners. We build competent diffusion language models at scale by first acquiring knowledge from massive data via masked language modeling pretraining thanks to their intrinsic connections. We then reprogram pretrained masked language models into diffusion language models via diffusive adaptation, wherein task-specific finetuning and instruction finetuning are explored to unlock their versatility in solving general language tasks. Experiments show that scaling diffusion language models consistently improves performance across downstream language tasks. We further discover that instruction finetuning can elicit zero-shot and few-shot in-context learning abilities that help tackle many unseen tasks by following natural language instructions, and show promise in advanced and challenging abilities such as reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
