arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.04264stat.MLcs.LG
收藏

解耦动态丰富性与表示学习:迈向实用测量

Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement

Oxford University(牛津大学);Seoul National University(首尔国立大学);Johns Hopkins University(约翰霍普金斯大学)

Yoonsoo Nam, Nayara Fonseca, Seok Hyeong Lee, Chris Mingard, Niclas Goring, Ouns El Harzli, Abdurrahman Hadi Erturk, Soufiane Hayou, Ard A. Louis

AI总结:

本文提出了一种计算高效且性能无关的丰富性度量标准,用于分析动态丰富性与表示学习之间的关系,并探讨了训练因素对丰富性的影响。

摘要:

动态特征变换(丰富 regime)并不总是与预测性能(更好的表示)一致,然而准确性常被用作丰富性的代理,限制了对其关系的分析。我们提出了一种计算高效、性能无关的丰富性度量标准,基于丰富动态的低秩偏差,该度量标准将神经坍塌作为特殊情况恢复。该度量标准在经验上比现有替代方案更稳定,并能捕捉已知的懒惰到丰富过渡(例如 grokking)而无需依赖准确性。我们进一步利用它来研究训练因素(例如学习率)与丰富性之间的关系,确认了已知的假设并突显了新的观察(例如批量归一化促进丰富动态)。还引入了基于特征分解的可视化方法以支持可解释性,共同提供了一种用于研究训练因素、动态和表示之间关系的诊断工具。

英文摘要:

Dynamic feature transformation (the rich regime) does not always align with predictive performance (better representation), yet accuracy is often used as a proxy for richness, limiting analysis of their relationship. We propose a computationally efficient, performance-independent metric of richness grounded in the low-rank bias of rich dynamics, which recovers neural collapse as a special case. The metric is empirically more stable than existing alternatives and captures known lazy-torich transitions (e.g., grokking) without relying on accuracy. We further use it to examine how training factors (e.g., learning rate) relate to richness, confirming recognized assumptions and highlighting new observations (e.g., batch normalization promotes rich dynamics). An eigendecomposition-based visualization is also introduced to support interpretability, together providing a diagnostic tool for studying the relationship between training factors, dynamics, and representations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑