微型自回归递归模型
Tiny Autoregressive Recursive Models
AI总结：
本文提出自回归TRM，评估其在小型自回归任务上的表现，发现两步细化基线表现强劲，但完整架构未见显著性能提升。
摘要：
微型递归模型（TRMs）最近在ARC-AGI上展示了显著的性能，证明了非常小的模型可以通过一个两步细化机制与大型基础模型竞争，该机制更新内部推理状态$z$和预测输出$y$。自然地，这种细化对于任何预测器都是有吸引力的，因此可以自然地想知道TRM机制是否可以有效地重新应用于自回归模型。然而，TRMs无法简单地与标准模型进行比较，因为它们缺乏因果预测结构，并且包含持久的潜在状态，这使得难以隔离特定的性能提升。在本文中，我们提出了自回归TRM，并在小型自回归任务上对其进行评估。为了了解其有效性，我们提出了一套模型，逐步将标准Transformer转换为微型自回归递归模型，在受控环境中固定块设计、令牌流和下一个令牌目标。在计算匹配的字符级算法任务上，我们意外发现有一些两层细化基线在表现上非常强劲。与预期相反，我们发现自回归TRM架构的完整版本没有可靠的性能提升。这些结果为更广泛的两步细化机制提供了潜在的前景，但同时也提醒人们不要将自回归TRM特定模型作为有前途的研究方向。
英文摘要：
Tiny Recursive Models (TRMs) have recently demonstrated remarkable performance on ARC-AGI, showing that very small models can compete against large foundation models through a two-step refinement mechanism that updates an internal reasoning state $z$ and the predicted output $y$. Naturally, such refinement is of interest for any predictor; it is therefore natural to wonder whether the TRM mechanism could be effectively re-adopted in autoregressive models. However, TRMs cannot be simply compared to standard models because they lack causal predictive structures and contain persistent latent states that make it difficult to isolate specific performance gains. In this paper, we propose the Autoregressive TRM and evaluate it on small autoregressive tasks. To understand its efficacy, we propose a suite of models that gradually transform a standard Transformer to a Tiny Autoregressive Recursive Model in a controlled setting that fixes the block design, token stream, and next-token objective. Across compute-matched experiments on character-level algorithmic tasks, we surprisingly find that there are some two-level refinement baselines that show strong performance. Contrary to expectations, we find no reliable performance gains from the full Autoregressive TRM architecture. These results offer potential promise for two-step refinement mechanisms more broadly but caution against investing in the autoregressive TRM-specific model as a fruitful research direction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
