AI 中文总结
本文提出 LoRA-Diffusion，将低秩分解应用于扩散式语言模型的去噪轨迹而非权重，引入轨迹级适配器等技术，在 SST-2 等数据集上取得优异性能，为扩散式语言模型提供参数高效微调框架。
AI 中文摘要
LoRA等参数高效微调方法已革新了大型自回归语言模型的适配方式，仅需极少可训练参数即可实现任务定制，但这类方法尚未成功扩展至扩散式语言模型——这类模型通过迭代去噪而非顺序 token 预测生成文本。本文提出 LoRA-Diffusion，一种参数高效微调方法，将低秩分解应用于去噪轨迹而非模型权重。与修改单个变换矩阵的权重型 LoRA 不同，该方法学习从噪声到输出的整个扩散路径的低秩扰动，引入了修改每个去噪步骤的轨迹级低秩适配器、跨扩散阶段的步骤自适应秩分配，以及允许推理时合并特定任务模块而无需重新训练的组合式多任务学习。在 SST-2、QNLI 和 MRPC 数据集上，本文报告了 5 个随机种子的 token 级去噪验证准确率，LoRA-Diffusion 在 SST-2 上达到最高平均性能，在 QNLI 和 MRPC 上表现强劲；联合多任务训练进一步显示，LoRA-Diffusion 在所有评估方法中达到最高 token 级准确率，与全微调相比，该方法减少了每个任务的存储需求，为扩散式语言模型建立了参数高效微调框架。
英文摘要
Parameter-efficient fine-tuning methods such as LoRA have transformed the adaptation of large autoregressive language models, enabling task-specific customization with substantially fewer trainable parameters. However, these methods have not been successfully extended to diffusion-based language models, which generate text through iterative denoising rather than sequential token prediction. We propose LoRA-Diffusion, a parameter-efficient fine-tuning approach that applies low-rank decomposition to the denoising trajectory instead of model weights. Unlike weight-based LoRA, which modifies individual transformation matrices, our method learns low-rank perturbations to the entire diffusion path from noise to output. We introduce trajectory-level low-rank adapters that modify each denoising step, step-adaptive rank allocation across diffusion phases, and compositional multi-task learning that allows merging task-specific modules at inference without retraining. On SST-2, QNLI, and MRPC, we report token-level denoising validation accuracy over five random seeds. LoRA-Diffusion achieves the highest mean performance on SST-2 and strong performance on QNLI and MRPC. Joint multi-task training further shows that LoRA-Diffusion achieves the highest token-level accuracy among the evaluated methods. The approach reduces per-task storage compared with full fine-tuning and establishes a parameter-efficient fine-tuning framework for diffusion language models.
