Thoughtbubbles: 一种用于潜在空间中并行思维的无监督方法
Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space
AI总结：
Thoughtbubbles是一种通过预训练学习并行自适应计算的转换器变体，能够在潜在空间中实现更高效的计算，提升模型的推理性能和零样本评估表现。
摘要：
当前在转换器中扩展推理时间计算的方法是训练模型在生成答案前先生成显式的思路链标记。虽然这些方法很强大，但它们的限制在于无法在预训练期间使用，并且只能依赖于依次生成的自然语言描述。在本工作中，我们提出了Thoughtbubbles，一种转换器变体，通过学习 fork 或删除残差流，在潜在空间中原生地进行并行自适应计算。因此，需要更多计算的标记可以在网络中间形成
英文摘要：
Current approaches for scaling inference-time compute in transformers train them to emit explicit chain-of-thought tokens before producing an answer. While these methods are powerful, they are limited because they cannot be applied during pretraining and rely solely on serially-generated, natural-language verbalization. In this work, we propose Thoughtbubbles, a transformer variant that natively performs parallel adaptive computation in latent space by learning to fork or delete residual streams. Thus, tokens requiring more computation can form a "bubble" of cloned residuals in the middle of the network. Crucially, this behavior is learned during pretraining with only language modeling loss. Using half of the training budget, Thoughtbubbles outperforms the perplexity and zero-shot evals of both standard decoder LMs and those using non-adaptive parallel computation approaches. These results hold across model sizes from 150M to 1.9B. Thoughtbubbles achieves competitive GSM8K results using half of the baseline's token budget. The implicit nature of our method enables models to begin learning adaptive computation at pretraining time, paving the way to unified train-time and test-time scaling behaviors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
