关于神经扩展定律的不变性与普遍性
Meta-Scaling Laws: On the Invariance and Generality of Neural Scaling
AI总结:
本文探讨如何建立通用的神经扩展定律,通过识别不变量,使定律能在不同领域间可靠传输,验证了在语言、视觉和语音任务中的有效性,并展示了跨领域应用。
摘要:
神经扩展定律建立了模型性能与数据或计算资源之间的可预测关系,为新领域和任务的资源分配提供关键指导。然而,此类定律最需要的地方也是最难获得的:为新模型任务对拟合需要昂贵的扫描,通常耗尽该定律旨在节省的计算预算。本文提出研究问题:如何开发通用的扩展定律?即在资源充足的源领域拟合一次,并可靠地传输到新领域,其中运行完整扫描不可行,这需要对何时以及为何扩展属性变化的根本理解。我们通过识别正确的不变量来解决这一问题:扩展定律在数据的双射(信息保持)变换下保持不变,并在非双射变换下以可预测、信息论基础的方式改变,这些变换降低了其信息分辨率ρ:一个定律在某一领域的单轴可以传输到另一个领域。我们验证了这一点在语言、视觉和语音中的有效性,并展示了两个跨领域应用:预测在电子健康记录上训练的语言模型的扩展,以及在不同噪声注入水平下的时间序列分类扩展,恢复数据扩展指数的误差在3%以内。
英文摘要:
Neural scaling laws establish a predictable relationship between model performance and data or compute, offering crucial guidance for resource allocation in new domains and tasks. Yet such laws are most needed precisely where they are hardest to obtain: fitting one for a new model and task pair demands expensive sweeps that typically exhaust the very compute budget the law is meant to economize. We study meta-scaling laws: relations that characterize when neural scaling behavior is preserved and predict how it changes under data transformations, enabling forecasts across domains when the shift can be approximated by a compatible transformation. We address this by identifying the right invariants: scaling laws are preserved under bijective (information-preserving) transformations of the data and modified in predictable, information-theoretically grounded ways under non-bijective transformations that lower its information resolution $ρ$: a single axis along which a law fit in one domain can be transported to another. We validate this across language, vision, and speech, and demonstrate two cross-domain applications: predicting scaling for language models trained on electronic health records from laws fit on general text, and predicting time-series classification scaling under varying levels of noise injection, recovering the data-scaling exponents to within $3\%$ error.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。