仅解码器还是编码器-解码器？将语言模型解释为正则化的编码器-解码器
Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder
AI总结：
本文通过正则化编码器-解码器结构对比分析仅解码器与编码器-解码器架构，揭示并理论分析了仅解码器模型中的注意力退化问题，提出部分注意力语言模型加以解决，在多项生成任务上验证了有效性。
摘要：
序列到序列（seq2seq）任务旨在基于给定的输入源序列生成目标序列。传统上，大多数seq2seq任务由Encoder-Decoder框架解决，该框架需要一个编码器来编码源序列，以及一个解码器来生成目标文本。最近，出现了许多将仅解码器语言模型直接应用于seq2seq任务的新方法。尽管在将语言模型应用于seq2seq任务方面取得了重大进展，但对仅解码器语言模型架构的有效性仍缺乏深入分析。本文旨在通过分析正则化的编码器-解码器结构，对编码器-解码器架构和仅解码器语言模型框架进行详细比较，从而填补这一空白。该结构旨在复制经典仅解码器语言模型中的所有行为，但具有编码器和解码器，使其更易于与经典的编码器-解码器结构进行比较。基于分析，我们揭示了语言模型中的注意力退化问题，即随着生成步数的增加，集中在源序列上的注意力越来越少。为了定量理解这一问题，我们对注意力输出相对于源输入进行了理论敏感性分析。基于我们的分析，我们提出了一种新颖的部分注意力语言模型来解决注意力退化问题。在机器翻译、摘要和数据到文本生成任务上的实验结果支持了我们的分析，并证明了我们提出模型的有效性。
英文摘要：
The sequence-to-sequence (seq2seq) task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by the Encoder-Decoder framework which requires an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task. Despite the significant advancements in applying language models to the seq2seq task, there is still a lack of thorough analysis on the effectiveness of the decoder-only language model architecture. This paper aims to address this gap by conducting a detailed comparison between the encoder-decoder architecture and the decoder-only language model framework through the analysis of a regularized encoder-decoder structure. This structure is designed to replicate all behaviors in the classical decoder-only language model but has an encoder and a decoder making it easier to be compared with the classical encoder-decoder structure. Based on the analysis, we unveil the attention degeneration problem in the language model, namely, as the generation step number grows, less and less attention is focused on the source sequence. To give a quantitative understanding of this problem, we conduct a theoretical sensitivity analysis of the attention output with respect to the source input. Grounded on our analysis, we propose a novel partial attention language model to solve the attention degeneration problem. Experimental results on machine translation, summarization, and data-to-text generation tasks support our analysis and demonstrate the effectiveness of our proposed model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
