arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.10107cs.CLcs.AI
收藏

用于可控扩散语言模型的量化嵌入向量

Quantized Embedding Vectors for Controllable Diffusion Language Models

Czech Technical University in Prague(捷克技术大学布拉格分校);Charles University(查理大学);University of Hong Kong(香港大学)

Cheng Kang, Xinye Chen, Yong Hu, Daniel Novak

AI总结:

提出量化嵌入可控扩散语言模型(QE-CDLM),通过量化重塑任务嵌入空间引入梯度控制器,提升模型可控性、可移植性与推理速度,实现更优困惑度与轻量级微调。

摘要:

提升扩散语言模型(DLMs)的可控性、可移植性和推理速度是自然语言生成中的关键挑战。尽管近期研究显示语言模型在复杂文本生成方面取得显著成功,但其对内存和计算能力的需求依然很高且未达预期,这自然导致模型可移植性低且不稳定。为缓解这些问题,已有许多成熟的神经网络量化方法被提出。为进一步增强其独立部署的可移植性,并改善由语言困惑度评估的稳定性,我们提出了一种名为量化嵌入可控扩散语言模型(QE-CDLM)的新方法。QE-CDLM基于近期成功的可控DLM,通过量化重塑特定任务的嵌入空间。这为生成任务引入了基于梯度的控制器,并获得了更稳定的中间潜在变量,从而自然带来加速收敛以及更好的可控性。此外,采用适配微调方法以减少可调权重。在五个具有挑战性的细粒度控制任务上的实验结果表明,QE-CDLM在质量和可行性上优于现有方法,实现了更好的困惑度和轻量级微调。

英文摘要:

Improving the controllability, portability, and inference speed of diffusion language models (DLMs) is a key challenge in natural language generation. While recent research has shown significant success in complex text generation with language models, the memory and computational power are still very demanding and fall short of expectations, which naturally results in low portability and instability for the models. To mitigate these issues, numerous well-established methods were proposed for neural network quantization. To further enhance their portability of independent deployment as well as improve their stability evaluated by language perplexity, we propose a novel approach called the Quantized Embedding Controllable Diffusion Language Model (QE-CDLM). QE-CDLM builds upon the recent successful controllable DLMs by remodeling the task-specific embedding space via quantization. This leads to a gradient-based controller for the generation tasks, and more stable intermediate latent variables are obtained, which naturally brings in an accelerated convergence as well as better controllability. Additionally, the adaption fine-tuning method is employed to reduce tunable weights. Experimental results on five challenging fine-grained control tasks demonstrate that QE-CDLM compares favorably to existing methods in terms of quality and feasibility, achieving better perplexity and lightweight fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑