arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.10074cs.LGcs.AI
收藏

边际在条件之前

Marginals Before Conditionals

School of Information Sciences University of Illinois Urbana-Champaign(信息科学学院伊利诺伊大学厄巴纳-香槟分校)

Mihir Sahasrabudhe

AI总结:

研究揭示神经网络中边际学习先于条件学习的现象,通过最小任务展示条件学习的动态过程及稳定性机制。

摘要:

我们构建了一个最小任务,以隔离神经网络中的条件学习:一个具有K折模糊性的映射,通过选择器标记z来解决,因此H(A | B) = log K而H(A | B, z) = 0。模型首先学习边际P(A | B),在恰好log K处形成一个平台,然后在尖锐的集体过渡中获得完整的条件。该平台有清晰的分解:高度= log K(由模糊性设定),持续时间= f(D)(由数据集大小D设定,而非K)。梯度噪声稳定了边际解:更高的学习率单调减缓过渡(在固定吞吐量下,3.6*跨7* η范围),并且批量大小减少会延迟逃脱,这与熵力对抗离开低梯度边际的趋势一致。内部,选择器路由头在平台期间形成,导致损失过渡约50%的等待时间。这是Papadopoulos等人[2024]所说的类型2方向不对称性,动态测量:我们跟踪从log K到零的超额风险,并表征什么稳定了它,什么触发了它的崩溃,以及它需要多长时间。

英文摘要:

We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* η range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑