arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.13660cs.LGcs.AI
收藏

预训练与测试时训练预算对ARC的影响:VARC与TRM的对照研究

Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM

Montefiore Institute of Electrical Engineering and Computer Science, University of Liège(列日大学蒙泰菲奥雷电气与计算机科学学院)

Adrien Deliege, Marc Van Droogenbroeck

AI总结:

本研究对照探究预训练与测试时训练预算对VARC、TRM模型在ARC任务上的性能影响,发现预算差异会改变模型表现,需报告可比预算下的结果。

摘要:

抽象与推理语料库(ARC)正越来越多地用于评估神经网络能否从少量演示中推断出特定任务的变换。近期的ARC模型在架构上存在差异,同时在更广泛的训练与评估协议(包括预训练数据、数据增强、测试时设置及优化预算)上也有所不同,这使得关于架构优势的结论存在争议。本研究在ARC-AGI-1的通用归纳协议下,探究受视觉启发的VARC模型与基于递归的TRM模型的性能如何受预训练和测试时训练(TTT)预算的影响。我们针对每个测试任务,将5种预训练预算与4种TTT预算进行交叉,每种预算指定任务变体数量与总样本呈现次数,以此定义受控训练预算。我们观察到,在所有设置中,增加TTT样本呈现次数可提升两种模型的标准ARC pass@2指标;增加TTT任务变体数量仅对TRM有持续帮助,而VARC在变体数量较少时往往表现更好;此外,VARC在无预训练时仍保持竞争力,而TRM从预训练中获益更多。这些结果表明,预训练与TTT预算会影响ARC评估,因此有必要报告可比预算下的结果。

英文摘要:

The Abstraction and Reasoning Corpus (ARC) is increasingly used to evaluate whether neural networks can infer task-specific transformations from few demonstrations. Recent ARC models differ in architecture, but also in broader training and evaluation protocols, including pretraining data, data augmentation, test-time setup, and optimization budgets, making conclusions about architecture benefits debatable. In this work, we study how the performance of the vision-inspired VARC and recursion-based TRM models is affected by pretraining and test-time training (TTT) budgets under a common inductive protocol on ARC-AGI-1. We cross five pretraining budgets with four TTT budgets for each test task. Each budget specifies a task-variant count and total sample presentations, defining the controlled training budgets. We observe that increasing TTT sample presentations improves the standard ARC pass@2 metric in all settings for both models. Increasing the TTT task-variant count consistently helps only TRM, whereas VARC often performs better with fewer variants. VARC also remains competitive without pretraining, while TRM benefits much more from it. These results show that pretraining and TTT budgets affect ARC evaluation and motivate reporting at comparable budgets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑