arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.09273cs.CL
收藏

Twist Decoding:多种发电机相互指导

Twist Decoding: Diverse Generators Guide Each Other

Tohoku University(东北大学);Allen Institute for AI(艾伦人工智能研究所);Yale University(耶鲁大学);University of Washington(华盛顿大学)

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Hao Peng, Ximing Lu, Dragomir Radev, Yejin Choi, Noah A. Smith

摘要:

许多语言生成模型现在可用于广泛的生成任务,包括机器翻译和摘要。组合这些不同的模型可能会导致进一步的进展,但集成生成模型在推理过程中具有挑战性:传统的集成方法(例如,浅融合)要求模型共享词汇表/标记化方案。我们介绍Twist解码,一个简单而通用的文本生成算法,在推理时受益于不同的模型。我们的方法不假设词汇表,标记化甚至生成顺序是共享的。我们对机器翻译和科学论文摘要的广泛评估表明,Twist解码在各种情况下都大大优于孤立解码的每个模型,包括特定领域和通用模型都可用的情况。扭曲解码也始终优于流行的重新排序启发式,其中来自一个模型的输出候选者被另一个重新评分。我们希望,我们的工作将鼓励研究人员和从业人员共同研究生成模型,而不仅仅是独立的,并寻求与现有模型互补的模型。我们的代码可在https://github.com/jungokasai/twist_decoding上获得。

英文摘要:

Many language generation models are now available for a wide range of generation tasks, including machine translation and summarization. Combining such diverse models may lead to further progress, but ensembling generation models is challenging during inference: conventional ensembling methods (e.g., shallow fusion) require that the models share vocabulary/tokenization schemes. We introduce Twist decoding, a simple and general text generation algorithm that benefits from diverse models at inference time. Our method does not assume the vocabulary, tokenization or even generation order is shared. Our extensive evaluations on machine translation and scientific paper summarization demonstrate that Twist decoding substantially outperforms each model decoded in isolation over various scenarios, including cases where domain-specific and general-purpose models are both available. Twist decoding also consistently outperforms the popular reranking heuristic where output candidates from one model are rescored by another. We hope that our work will encourage researchers and practitioners to examine generation models collectively, not just independently, and to seek out models with complementary strengths to the currently available models. Our code is available at https://github.com/jungokasai/twist_decoding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑