发表机构
Aston University（阿斯顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过参考模拟器发现动态张量重物化策略存在两种确定性预算敏感缺陷：LSTM轨迹上的模式切换导致高达7.3倍开销差异，以及ResNet-32上的可行性反转，并归因于联合大小-陈旧性评分项。
AI 中文摘要
我们报告了动态张量重物化（DTR）中细粒度、确定性的不稳定性。DTR是一种用于内存受限的DNN训练中的在线逐出策略。我们使用公共执行轨迹，在参考DTR模拟器（simrd）上进行了测量。在一条LSTM轨迹上，内存预算相差无约束峰值内存的0.10%时，会选择开销差异高达7.3倍的快速和慢速执行模式；慢速模式由同一存储的广泛重复重新逐出驱动（每次存储的逐出次数从1.33上升到8.27，而被逐出的不同存储集合基本不变：5233对5236，两个集合的Jaccard重叠度为0.999）。在一条ResNet-32轨迹上，精细的预算扫描揭示了一种确定性的可行性反转：运行在比率0.101时可行，在0.102-0.106范围内不可行（内存耗尽），从0.107开始再次可行。我们将内存耗尽的直接原因追溯到完全固定的递归重物化前沿，该前沿在每一个可逐出张量被逐出后超出预算。使用DTR作者自身变体的消融实验表明，观察到的LSTM不稳定性与联合大小-陈旧性评分项有关。我们认为这些至少是两种不同的预算敏感病理，而非单一机制，并区分了已证明的内容与仍属假设的内容。所有结果均涉及参考模拟器；在生产运行时中的复现是未来的工作。代码、工具和原始结果随本预印本一同提供。
英文摘要
We report fine-grained, deterministic instability in Dynamic Tensor Rematerialization (DTR), an online eviction policy for memory-constrained DNN training, measured on the reference DTR simulator (simrd) using public execution traces. On an LSTM trace, memory budgets differing by 0.10% of unconstrained peak memory select fast and slow execution regimes whose overheads differ by as much as 7.3x; the slow regime is driven by broadly repeated re-eviction of the same storages (evictions per storage rise from 1.33 to 8.27 while the set of distinct evicted storages is essentially unchanged: 5,233 vs 5,236, with the two sets overlapping at Jaccard 0.999). On a ResNet-32 trace, a fine budget sweep reveals a deterministic feasibility inversion: the run is feasible at ratio 0.101, infeasible (OOM) across 0.102-0.106, and feasible again from 0.107. We trace the immediate cause of the OOM to a fully pinned recursive rematerialization frontier that exceeds the budget after every evictable tensor has been evicted. Ablations using the DTR authors' own variants implicate the joint size-staleness scoring term in the observed LSTM instability. We argue these are at least two distinct budget-sensitive pathologies rather than one mechanism, and we separate what is demonstrated from what remains hypothesised. All results concern the reference simulator; reproduction in a production runtime is future work. Code, instrumentation, and raw results accompany this preprint.
Comments6 pages, 5 tables, 2 figures. Code and data: https://github.com/lonewolf15116/dtr-regime-switching
