arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.00894cs.LGcs.CL
收藏

何时思考:通过测试时间训练实现的自适应计算分配

When to Ponder: Adaptive Compute Allocation for Code Generation via Test-Time Training

Dongpae High School, Paju, Gyeonggi-do, Republic of Korea(京畿道帕久郡东垈高中)

Gihyeon Sim

AI总结:

通过测试时间训练实现自适应计算分配,利用自监督重建损失选择性更新,提升代码语言建模性能

摘要:

大型语言模型对所有输入应用均匀的计算,无论难度如何。我们提出了PonderTTT,一种使用TTT层的自监督重建损失的门控策略,以选择性地触发测试时间训练(TTT)更新。门控决策本身是无训练的——不需要任何学习的分类器或辅助网络;仅需在未标记数据上初始校准一个单一标量阈值,并通过EMA持续适应以维持目标更新率。我们在GPT-2模型(124M到1.5B)上进行的实验,针对代码语言建模(The Stack v2,教师强制困惑度)显示,这种信号是推理兼容的,不需要真实标签。我们的重建门控在完全无训练的情况下实现了82-89%的Oracle恢复率,显著优于随机跳过基线(在OOD语言上损失低至16%)

英文摘要:

Large language models apply uniform computation to all inputs, regardless of difficulty. We propose PonderTTT, a gating strategy using the TTT layer's self-supervised reconstruction loss to selectively trigger Test-Time Training (TTT) updates. The gating decision itself is training-free--requiring no learned classifier or auxiliary networks; only a single scalar threshold is initially calibrated on unlabeled data and continuously adapted via EMA to maintain target update rates. Our experiments with GPT-2 models (124M to 1.5B) on code language modeling (The Stack v2, teacher-forced perplexity) demonstrate that this signal is inference-compatible, requiring no ground-truth labels. Our Reconstruction Gating achieves 82-89% Oracle Recovery while being fully training-free, significantly outperforming Random Skip baselines (up to 16% lower loss on OOD languages).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑