数据应该教什么？跨越电路、存储与使用的瓶颈迁移
What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use
AI总结：
本研究通过电路视角识别语言模型训练中的三个瓶颈，提出诊断到数据原则，并通过实验证明针对瓶颈的监督变化能改善学习与决策。
AI中文摘要：
在训练的特定时刻，数据应该教语言模型什么？电路视角揭示了三个不同的瓶颈：形成计算、使其所需内容可用，以及在可用路径中进行选择。一个共享的诊断到数据原则将它们联系起来：定位缺失的操作，保留其因果关系，变化携带捷径的上下文，并重新审计残差。对形成敏感的选择和先决条件排序加速了绑定-匹配-传输路径；来自同一训练多集的早期简短前缀在100B个令牌中保持了验证优势。可用性反事实随后区分了写入内容与调用可用记忆，而配对监督和上下文机会排序改善了匹配路由决策和长上下文答案可能性。一个连续的350M模型实验在相同事实上连接了三种干预：早期电路训练改善了后续学习，完整的序列在从使用教学中保留的事实上优于阶段替换控制。独立查询表面和反对源决策暴露了条件仲裁作为剩余的边界。总之，这些结果显示了为什么限制操作的变化需要监督的变化，而不仅仅是困难示例的新排序。
英文摘要：
What should data teach a language model at a particular point in training? A circuit view reveals three distinct bottlenecks: forming a computation, making its required content available, and selecting among available routes. A shared diagnosis-to-data principle connects them: localize the missing operation, preserve its causal relation, vary shortcut-bearing context, and re-audit the residual. Formation-sensitive selection and prerequisite ordering accelerate a binding-matching-transport path; a brief early prefix from the same training multiset retains a validation advantage through 100B tokens. Availability counterfactuals then distinguish writing content from invoking available memory, while paired supervision and context-opportunity ranking improve matched route decisions and long-context answer likelihood. A continuous 350M-model experiment connects the three interventions on the same facts: early circuit training improves subsequent learning, and the complete sequence outperforms stage-replacement controls on facts withheld from Use teaching. Independent query surfaces and opposed-source decisions expose conditional arbitration as the remaining frontier. Together, these results show why a change in the limiting operation calls for a change in supervision, not merely a new ranking of difficult examples.
