arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.05424cs.CL
收藏

呼吁澄清 Beam Search:它如何工作以及何时停止

A Call for Clarity in Beam Search: How It Works and When It Stops

Toyota Technological Institute at Chicago(芝加哥丰田工业大学);Tohoku University(东北大学);RIKEN(日本理化学研究所);Allen Institute for AI(艾伦人工智能研究院);Yale University(耶鲁大学);University of Washington(华盛顿大学)

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Dragomir Radev, Yejin Choi, Noah A. Smith

AI总结:

本文揭示常用 beam decoding 的先到先得停止机制,提出仅改一行代码的 patience factor,以可忽略开销提升摘要与翻译性能,并呼吁明确实现细节。

摘要:

使用 beam search 进行文本生成已在广泛应用中被证明是成功的。我们指出,尽管在文献中很大程度上被忽视,但常用的 beam decoding 实现(例如 Hugging Face Transformers 和 fairseq)采用了一种先到先得的启发式方法:它在各个时间步保留一组已完成的序列,并在该集合的大小达到 beam size 时停止。基于这一发现,我们引入了 patience factor,这是对该 beam decoding 实现的一个简单修改,它泛化了停止准则,并为搜索深度提供了灵活性。实验结果表明,在新闻文本摘要和多种语言对的机器翻译任务上,调整这一 patience factor 可以提升强大预训练模型的解码性能,而推理速度的降低可以忽略不计。我们的方法只修改一行代码,因此可以很容易地纳入任何实现。此外,我们发现不同版本的 beam decoding 会在摘要任务中导致巨大的性能差异,这表明在研究工作中有必要明确说明 beam search 的实现细节。我们的代码将在论文发表后公开。

英文摘要:

Text generation with beam search has proven successful in a wide range of applications. We point out that, though largely overlooked in the literature, the commonly-used implementation of beam decoding (e.g., Hugging Face Transformers and fairseq) uses a first come, first served heuristic: it keeps a set of already completed sequences over time steps and stops when the size of this set reaches the beam size. Based on this finding, we introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search. Empirical results demonstrate that adjusting this patience factor improves decoding performance of strong pretrained models on news text summarization and machine translation over diverse language pairs, with a negligible inference slowdown. Our approach only modifies one line of code and can be thus readily incorporated in any implementation. Further, we find that different versions of beam decoding result in large performance differences in summarization, demonstrating the need for clarity in specifying the beam search implementation in research work. Our code will be available upon publication.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑