探索LLM用于单步文本生成的隐藏能力
Exploring the Hidden Capacity of LLMs for One-Step Text Generation
AI总结：
本研究发现冻结的LLMs仅通过两个学习到的嵌入即可在单次前向传递中生成数百个准确token，揭示了其原生多token生成能力，为克服自回归解码瓶颈提供了新途径。
摘要：
最近一项研究表明，大型语言模型（LLMs）能够仅通过一个训练好的输入嵌入进行自回归生成，从而重建长达数千个token的惊人长文本。在本研究中，我们探讨自回归解码对于这种重建是否必不可少。我们表明，当仅提供两个学习到的嵌入时，冻结的LLMs能够在一次token并行前向传递中生成数百个准确的token。这揭示了自回归LLMs一种令人惊讶且尚未充分探索的多token生成能力。我们检查了这些嵌入并表征了它们编码的信息。我们还通过实证表明，尽管这些表示对于给定文本并不唯一，但它们在嵌入空间中形成了连通的局部区域——这表明训练实用编码器具有潜力。此类表示的存在暗示，多token生成可能通过学习到的输入编码器在现成的LLMs中原生实现，从而消除繁重的重新训练，有助于克服自回归解码的基本瓶颈，同时重用已训练的模型。
英文摘要：
A recent study showed that large language models (LLMs) can reconstruct surprisingly long texts - up to thousands of tokens - via autoregressive generation from just one trained input embedding. In this work, we explore whether autoregressive decoding is essential for such reconstruction. We show that frozen LLMs can generate hundreds of accurate tokens in just one token-parallel forward pass, when provided with only two learned embeddings. This reveals a surprising and underexplored multi-token generation capability of autoregressive LLMs. We examine these embeddings and characterize the information they encode. We also empirically show that, although these representations are not unique for a given text, they form connected and local regions in embedding space - suggesting the potential to train a practical encoder. The existence of such representations hints that multi-token generation may be natively accessible in off-the-shelf LLMs via a learned input encoder, eliminating heavy retraining and helping to overcome the fundamental bottleneck of autoregressive decoding while reusing already-trained models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
