arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.07438cs.CLcs.AI
收藏

自回归语言生成的可处理控制

Tractable Control for Autoregressive Language Generation

University of California, Los Angeles(加州大学洛杉矶分校)

Honghua Zhang, Meihua Dang, Nanyun Peng, Guy Van den Broeck

AI总结:

针对自回归语言模型生成受约束文本的不可处理问题,提出利用可处理概率模型(TPMs)施加词汇约束的GeLaTo框架,在CommonGen等基准上取得最先进性能。

摘要:

尽管自回归大型语言模型在文本生成方面取得了成功,但生成满足复杂约束的文本仍然是一个重大挑战:即使对于最简单的词汇约束α,从条件分布Pr(text | α)中采样也是不可处理的。为了克服这一挑战,我们提出使用可处理概率模型(TPMs)在自回归文本生成模型中施加词汇约束,我们称之为GeLaTo(具有可处理约束的语言生成)。为了展示该框架的有效性,我们使用蒸馏隐马尔可夫模型,其中我们可以高效计算Pr(text | α),来引导GPT2的自回归生成。GeLaTo在约束文本生成的挑战性基准(如CommonGen)上取得了最先进的性能,大幅超越了各种强基线。我们的工作不仅为控制大型语言模型开辟了新途径,也激励了更具表达力的TPMs的发展。

英文摘要:

Despite the success of autoregressive large language models in text generation, it remains a major challenge to generate text that satisfies complex constraints: sampling from the conditional distribution ${\Pr}(\text{text} | α)$ is intractable for even the simplest lexical constraints $α$. To overcome this challenge, we propose to use tractable probabilistic models (TPMs) to impose lexical constraints in autoregressive text generation models, which we refer to as GeLaTo (Generating Language with Tractable Constraints). To demonstrate the effectiveness of this framework, we use distilled hidden Markov models, where we can efficiently compute ${\Pr}(\text{text} | α)$, to guide autoregressive generation from GPT2. GeLaTo achieves state-of-the-art performance on challenging benchmarks for constrained text generation (e.g., CommonGen), beating various strong baselines by a large margin. Our work not only opens up new avenues for controlling large language models but also motivates the development of more expressive TPMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑