扩散如何影响预训练语言模型在分布外数据上的表现？
How Does Diffusion Influence Pretrained Language Models on Out-of-Distribution Data?
AI总结：
本文通过重建损失分析扩散模型对预训练语言模型分布外鲁棒性的影响，发现微调降低重建能力，但扩散检测OOD样本达最优，准确率提升18%。
摘要：
基于Transformer的预训练语言模型（PLMs）在现代自然语言处理中取得了巨大成功。PLMs的一个重要优势是良好的分布外（OOD）鲁棒性。近年来，扩散模型吸引了大量工作将扩散应用于PLMs。然而，扩散如何影响PLMs在OOD数据上的表现仍未被充分探索。扩散模型的核心是一个前向扩散过程，该过程逐步向输入添加高斯噪声，以及一个反向去噪过程，该过程去除噪声。带噪声输入的重建是扩散模型的基本能力。我们通过测量重建损失直接分析OOD鲁棒性，包括测试重建OOD数据的能力以及检测OOD样本的能力。实验通过分析八个数据集上的不同训练参数和数据统计特征进行。结果表明，使用扩散对PLMs进行微调会降低其在OOD数据上的重建能力。比较还表明，扩散模型能够有效检测OOD样本，在大多数数据集上实现了最先进的性能，绝对准确率提升高达18%。这些结果表明，扩散降低了PLMs的OOD鲁棒性。
英文摘要：
Transformer-based pretrained language models (PLMs) have achieved great success in modern NLP. An important advantage of PLMs is good out-of-distribution (OOD) robustness. Recently, diffusion models have attracted a lot of work to apply diffusion to PLMs. It remains under-explored how diffusion influences PLMs on OOD data. The core of diffusion models is a forward diffusion process which gradually applies Gaussian noise to inputs, and a reverse denoising process which removes noise. The noised input reconstruction is a fundamental ability of diffusion models. We directly analyze OOD robustness by measuring the reconstruction loss, including testing the abilities to reconstruct OOD data, and to detect OOD samples. Experiments are conducted by analyzing different training parameters and data statistical features on eight datasets. It shows that finetuning PLMs with diffusion degrades the reconstruction ability on OOD data. The comparison also shows that diffusion models can effectively detect OOD samples, achieving state-of-the-art performance in most of the datasets with an absolute accuracy improvement up to 18%. These results indicate that diffusion reduces OOD robustness of PLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
