DiNADO：用于控制语言模型的范数解耦神经分解预言机
DiNADO: Norm-Disentangled Neurally-Decomposed Oracles for Controlling Language Models
AI总结：
本文提出DiNADO，通过解耦近似预言机R值的逐步全局范数，解决原始NADO的梯度消失、容量受限等问题，使其可与LoRA等微调方法结合，在正式性控制和CommonGen任务上验证了改进效果。
摘要：
神经分解预言机（NADO）是一种用于大语言模型可控生成的强大方法。其设计目标是在避免灾难性遗忘的同时，以合理的建模能力实现向熵最大化闭式最优解的保证收敛。尽管取得了成功，但在将NADO应用于广泛场景时仍面临若干挑战。原始NADO在低概率控制信号下存在梯度消失问题，并且高度依赖正则化来满足贝尔曼方程的随机版本。此外，NADO的原始实现引入了若干额外的Transformer层，其容量有限，尤其是与LoRA等其他基于微调的模型适配方法相比。本文提出了一种改进的NADO算法，即DiNADO（范数解耦神经分解预言机），该算法通过对所有潜在下一token的近似预言机R值进行逐步全局范数解耦来提升NADO算法的性能，从而使DiNADO能够与LoRA等微调方法相结合。我们从经验和理论两方面深入讨论了DiNADO如何实现更好的容量、稳定性和灵活性。在机器翻译中的正式性控制以及词汇约束生成任务CommonGen上的实验证明了这些改进的重要性。
英文摘要：
NeurAlly-Decomposed Oracle (NADO) is a powerful approach for controllable generation with large language models. It is designed to avoid catastrophic forgetting while achieving guaranteed convergence to an entropy-maximized closed-form optimal solution with reasonable modeling capacity. Despite the success, several challenges arise when apply NADO to a wide range of scenarios. Vanilla NADO suffers from gradient vanishing for low-probability control signals and is highly reliant on a regularization to satisfy the stochastic version of Bellman equation. In addition, the vanilla implementation of NADO introduces a few additional transformer layers, suffering from a limited capacity especially compared to other finetune-based model adaptation methods like LoRA. In this paper, we propose a improved version of the NADO algorithm, namely DiNADO (norm-Disentangled NeurAlly-Decomposed Oracles), which improves the performance of the NADO algorithm through disentangling the step-wise global norm over the approximated oracle $R$-value for all potential next-tokens, allowing DiNADO to be combined with finetuning methods like LoRA. We discuss in depth how DiNADO achieves better capacity, stability and flexibility with both empirical and theoretical results. Experiments on formality control in machine translation and the lexically constrained generation task CommonGen demonstrates the significance of the improvements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
