arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.10474cs.CL
收藏

ByGPT5:使用无 token 语言模型的端到端风格条件诗歌生成

ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models

Bielefeld University(比勒费尔德大学)

Jonas Belouadi, Steffen Eger

AI总结:

本文提出无 token 的仅解码器语言模型 ByGPT5,通过在带风格标注的英德四行诗语料上预训练与微调,实现端到端风格条件诗歌生成,并在性能、参数效率和抗记忆方面优于多个基线。

摘要:

最先进的诗歌生成系统通常很复杂。它们要么由特定任务的模型流水线组成,以人工创建约束的形式纳入先验知识,要么两者兼有。相比之下,端到端模型不会承担必须建模先验知识的开销,并且能够仅从数据中学习诗歌的细微差别,从而降低所需的人工监督程度。在这项工作中,我们研究了以押韵、格律和头韵等风格为条件的端到端诗歌生成。我们识别并解决了训练数据不足以及分词算法不匹配这些以往尝试可能存在的局限。特别是,我们成功预训练了 ByGPT5,这是一种新型的无 token、仅解码器语言模型,并在一个大型自定义语料库上对其进行微调,该语料库由带有我们所定义风格标注的英语和德语四行诗组成。我们表明,ByGPT5 优于 mT5、ByT5、GPT-2 和 ChatGPT 等其他模型,同时具有更高的参数效率,并且与人类相比表现良好。此外,我们分析了它的运行时性能,并证明它不容易发生记忆。我们将代码、模型和数据集公开发布。

英文摘要:

State-of-the-art poetry generation systems are often complex. They either consist of task-specific model pipelines, incorporate prior knowledge in the form of manually created constraints, or both. In contrast, end-to-end models would not suffer from the overhead of having to model prior knowledge and could learn the nuances of poetry from data alone, reducing the degree of human supervision required. In this work, we investigate end-to-end poetry generation conditioned on styles such as rhyme, meter, and alliteration. We identify and address lack of training data and mismatching tokenization algorithms as possible limitations of past attempts. In particular, we successfully pre-train ByGPT5, a new token-free decoder-only language model, and fine-tune it on a large custom corpus of English and German quatrains annotated with our styles. We show that ByGPT5 outperforms other models such as mT5, ByT5, GPT-2 and ChatGPT, while also being more parameter efficient and performing favorably compared to humans. In addition, we analyze its runtime performance and demonstrate that it is not prone to memorization. We make our code, models, and datasets publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑