关于用于一步文本重建的原生标记所编码的语义和语法信息
On the Semantic and Syntactic Information Encoded in Proto-Tokens for One-Step Text Reconstruction
AI总结：
本文研究了原生标记在一步文本重建中的语义和语法信息编码，通过实验分析了其稳定性及正则化方法，支持了非自回归序列到序列系统的可行性。
摘要：
自回归大型语言模型（LLMs）逐个生成文本标记，需要n次前向传递来生成长度为n的序列。最近的工作《探索LLMs的潜在容量以实现一步文本重建》（Mezentsev和Oseledets）表明，冻结的LLMs可以在单次前向传递中仅用两个学习得到的原生标记重建数百个标记，这表明了一条超越自回归范式的路径。在本文中，我们研究这些原生标记编码了什么信息，以及它们在重建和受控约束下的行为。我们进行了一系列实验，旨在解构两个原生标记中的语义和语法内容，分析e标记的稳定性属性，并在重建过程中可视化对e标记的注意力模式。最后，我们测试了两种正则化方案，用于“施加”语义结构到e标记上，包括基于锚点的损失和关系蒸馏目标。我们的结果表明，在标准优化下，m标记比e标记更能捕捉语义信息；基于锚点的约束与重建准确性之间存在显著的权衡；而关系蒸馏可以将批量级别的语义关系转移到原生标记空间中，而不牺牲重建质量，支持了未来非自回归序列到序列系统在预测原生标记作为中间表示方面的可行性。
英文摘要：
Autoregressive large language models (LLMs) generate text token-by-token, requiring n forward passes to produce a sequence of length n. Recent work, Exploring the Latent Capacity of LLMs for One-Step Text Reconstruction (Mezentsev and Oseledets), shows that frozen LLMs can reconstruct hundreds of tokens from only two learned proto-tokens in a single forward pass, suggesting a path beyond the autoregressive paradigm. In this paper, we study what information these proto-tokens encode and how they behave under reconstruction and controlled constraints. We perform a series of experiments aimed at disentangling semantic and syntactic content in the two proto-tokens, analyzing stability properties of the e-token, and visualizing attention patterns to the e-token during reconstruction. Finally, we test two regularization schemes for "imposing" semantic structure on the e-token using teacher embeddings, including an anchor-based loss and a relational distillation objective. Our results indicate that the m-token tends to capture semantic information more strongly than the e-token under standard optimization; anchor-based constraints trade off sharply with reconstruction accuracy; and relational distillation can transfer batch-level semantic relations into the proto-token space without sacrificing reconstruction quality, supporting the feasibility of future non-autoregressive seq2seq systems that predict proto-tokens as an intermediate representation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
