arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.01850cs.CL
收藏

多约束条件下连续优化的受控文本生成

Controlled Text Generation as Continuous Optimization with Multiple Constraints

Carnegie Mellon University(卡内基梅隆大学);Google Research(谷歌研究院);University of Washington(华盛顿大学)

Sachin Kumar, Eric Malmi, Aliaksei Severyn, Yulia Tsvetkov

摘要:

随着大规模语言模型预训练推动了文本生成的发展,最近的工作已经转向控制这些模型生成的文本的属性。虽然通过微调来修改预训练模型仍然是流行的方法,但它会产生显著的计算成本,并且由于缺乏适当的数据而不可行。作为替代方案,我们提出了MuCoCO -一种灵活的模块化算法,用于从预训练模型进行可控推理。我们制定的解码过程作为一个优化问题,它允许多个属性,我们的目标是控制容易被纳入作为可微约束的优化。通过将这种离散优化放松为连续优化,我们利用拉格朗日乘子和基于梯度下降的技术来生成所需的文本。我们评估了我们的方法在可控的机器翻译和风格转移与多个副本级别的属性,并观察到显着的改进基线。

英文摘要:

As large-scale language model pretraining pushes the state-of-the-art in text generation, recent work has turned to controlling attributes of the text such models generate. While modifying the pretrained models via fine-tuning remains the popular approach, it incurs a significant computational cost and can be infeasible due to lack of appropriate data. As an alternative, we propose MuCoCO -- a flexible and modular algorithm for controllable inference from pretrained models. We formulate the decoding process as an optimization problem which allows for multiple attributes we aim to control to be easily incorporated as differentiable constraints to the optimization. By relaxing this discrete optimization to a continuous one, we make use of Lagrangian multipliers and gradient-descent based techniques to generate the desired text. We evaluate our approach on controllable machine translation and style transfer with multiple sentence-level attributes and observe significant improvements over baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑