An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation
Meituan(美团);Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Xuancheng Huang, Zijun Liu, Peng Li, Tao Li, Maosong Sun, Yang Liu
备注:
long paper, accepted by ACL 2023 (main conference)
Recently, multi-aspect controllable text generation that controls the generated text in multiple aspects (e.g., sentiment, topic, and keywords) has attracted increasing attention. Although methods based on parameter efficient tuning like prefix-tuning could achieve multi-aspect controlling in a plug-and-play way, the mutual interference of multiple prefixes leads to significant degeneration of constraints and limits their extensibility to training-time unseen aspect combinations. In this work, we provide a theoretical lower bound for the interference and empirically found that the interference grows with the number of layers where prefixes are inserted. Based on these analyses, we propose using trainable gates to normalize the intervention of prefixes to restrain the growing interference. As a result, controlling training-time unseen combinations of aspects can be realized by simply concatenating corresponding plugins such that new constraints can be extended at a lower cost. In addition, we propose a unified way to process both categorical and free-form constraints. Experiments on text generation and machine translation demonstrate the superiority of our approach over baselines on constraint accuracy, text quality, and extensibility.
Controllable Text Generation with Residual Memory Transformer
School of Computer Science & Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系)
MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学)
DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);ByteDance Inc.(字节跳动公司)