arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.09387cs.CLcs.AI
收藏

一种用于多方面可控文本生成的可扩展即插即用方法

An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation

Meituan(美团);Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Xuancheng Huang, Zijun Liu, Peng Li, Tao Li, Maosong Sun, Yang Liu

AI总结:

本文针对多方面可控文本生成中多前缀相互干扰的问题,提出基于可训练门控归一化前缀干预的即插即用方法,实现训练时未见方面组合的低成本扩展,并统一处理类别型与自由形式约束,在约束准确率、文本质量和可扩展性上优于基线。

摘要:

近年来,在多个方面(如情感、主题和关键词)控制生成文本的多方面可控文本生成受到越来越多的关注。尽管基于参数高效微调(如前缀微调)的方法能够以即插即用的方式实现多方面控制,但多个前缀之间的相互干扰会导致约束显著退化,并限制其对训练时未见方面组合的可扩展性。本文给出了干扰的理论下界,并通过实验发现干扰会随着插入前缀的层数增加而增大。基于这些分析,我们提出使用可训练门控来归一化前缀的干预,以抑制不断增长的干扰。由此,只需简单拼接相应插件即可实现对训练时未见方面组合的控制,从而以更低成本扩展新的约束。此外,我们提出了一种统一方法来同时处理类别型约束和自由形式约束。在文本生成和机器翻译上的实验表明,我们的方法在约束准确率、文本质量和可扩展性方面均优于基线方法。

英文摘要:

Recently, multi-aspect controllable text generation that controls the generated text in multiple aspects (e.g., sentiment, topic, and keywords) has attracted increasing attention. Although methods based on parameter efficient tuning like prefix-tuning could achieve multi-aspect controlling in a plug-and-play way, the mutual interference of multiple prefixes leads to significant degeneration of constraints and limits their extensibility to training-time unseen aspect combinations. In this work, we provide a theoretical lower bound for the interference and empirically found that the interference grows with the number of layers where prefixes are inserted. Based on these analyses, we propose using trainable gates to normalize the intervention of prefixes to restrain the growing interference. As a result, controlling training-time unseen combinations of aspects can be realized by simply concatenating corresponding plugins such that new constraints can be extended at a lower cost. In addition, we propose a unified way to process both categorical and free-form constraints. Experiments on text generation and machine translation demonstrate the superiority of our approach over baselines on constraint accuracy, text quality, and extensibility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑