LoopFormer: 基于快捷键调制的弹性深度循环变换器用于潜在推理
LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
AI总结：
LoopFormer通过弹性深度循环变换器实现基于快捷键调制的潜在推理，具备适应不同计算预算的鲁棒性与扩展性。
摘要：
循环变换器已经涌现出作为在语言领域进行推理的高效且强大的模型类别。最近的研究表明，这些模型在算法和推理任务上实现了强大的性能，表明循环架构具有对潜在推理的归纳偏差。然而，先前的方法在训练和推理过程中固定循环迭代次数，留下了这些模型是否能在可变计算预算下灵活调整计算深度的问题。我们介绍了LoopFormer，一种在可变长度轨迹上训练的循环变换器，以实现预算条件下的推理。我们的核心贡献是一种快捷键一致性训练方案，它对不同长度的轨迹进行对齐，确保较短的循环产生信息丰富的表示，而较长的循环则继续优化这些表示。LoopFormer将每个循环条件于当前时间和步长大小，使表示在不同长度的轨迹上保持一致地演变，而不是漂移或停滞。实证结果表明，即使在严格的计算约束下，LoopFormer在语言建模和推理基准上也表现出鲁棒性，并且能够随着额外预算的增加而扩展。这些结果表明，循环变换器本质上适合于适应性语言建模，为可控和预算感知的大语言模型开辟了道路。
英文摘要：
Looped Transformers have emerged as an efficient and powerful class of models for reasoning in the language domain. Recent studies show that these models achieve strong performance on algorithmic and reasoning tasks, suggesting that looped architectures possess an inductive bias toward latent reasoning. However, prior approaches fix the number of loop iterations during training and inference, leaving open the question of whether these models can flexibly adapt their computational depth under variable compute budgets. We introduce LoopFormer, a looped Transformer trained on variable-length trajectories to enable budget-conditioned reasoning. Our core contribution is a shortcut-consistency training scheme that aligns trajectories of different lengths, ensuring that shorter loops yield informative representations while longer loops continue to refine them. LoopFormer conditions each loop on the current time and step size, enabling representations to evolve consistently across trajectories of varying length rather than drifting or stagnating. Empirically, LoopFormer demonstrates robust performance on language modeling and reasoning benchmarks even under aggressive compute constraints, while scaling gracefully with additional budget. These results show that looped Transformers are inherently suited for adaptive language modeling, opening a path toward controllable and budget-aware large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
