arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.26697cs.CLcs.AI
收藏

手动解码的终结:迈向真正的端到端语言模型

The End of Manual Decoding: Towards Truly End-to-End Language Models

Tencent AI Lab(腾讯AI实验室)

Zhichao Wang, Dongyang Ma, Xinting Huang, Deng Cai, Tian Lan, Jiahao Xu, Haitao Mi, Xiaoying Tang, Yan Wang

AI总结:

本文提出AutoDeco架构,通过在标准transformer上增加轻量级头部动态预测温度和top-p,实现真正的端到端LLM生成,在八个基准上表现优异,并展现出基于指令控制解码的涌现能力。

摘要:

LLM的“端到端”标签是一个误称。在实践中,它们依赖于不可微分的解码过程,需要对温度和top-p等超参数进行费力的手动调优。本文引入了AutoDeco,这是一种新颖的架构,通过学习控制自身的解码策略来实现真正的“端到端”生成。我们用轻量级头部增强了标准transformer,这些头部在每个步骤动态预测特定于上下文的温度和top-p值,以及下一个token的logits。这种方法将解码转化为参数化的token级过程,允许模型在单次前向传递中自我调节其采样策略。通过在八个基准上进行的大量实验,我们证明AutoDeco不仅显著优于默认解码策略,而且实现了与从“黑客攻击测试集”派生的oracle调优基线相当的性能——这是任何静态方法的实际上限。至关重要的是,我们发现了基于指令的解码控制的一种涌现能力:模型学会了解释自然语言命令(例如,“以低随机性生成”),并逐个token地调整其预测的温度和top-p,为可引导和交互式LLM解码开辟了新范式。

英文摘要:

The "end-to-end" label for LLMs is a misnomer. In practice, they depend on a non-differentiable decoding process that requires laborious, hand-tuning of hyperparameters like temperature and top-p. This paper introduces AutoDeco, a novel architecture that enables truly "end-to-end" generation by learning to control its own decoding strategy. We augment the standard transformer with lightweight heads that, at each step, dynamically predict context-specific temperature and top-p values alongside the next-token logits. This approach transforms decoding into a parametric, token-level process, allowing the model to self-regulate its sampling strategy within a single forward pass. Through extensive experiments on eight benchmarks, we demonstrate that AutoDeco not only significantly outperforms default decoding strategies but also achieves performance comparable to an oracle-tuned baseline derived from "hacking the test set"-a practical upper bound for any static method. Crucially, we uncover an emergent capability for instruction-based decoding control: the model learns to interpret natural language commands (e.g., "generate with low randomness") and adjusts its predicted temperature and top-p on a token-by-token basis, opening a new paradigm for steerable and interactive LLM decoding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑