arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22617cs.LG
收藏

语义管预测:通过JEPA超越LLM数据效率

Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA

Hai Huang, Yann LeCun, Randall Balestriero

AI总结:

本文提出语义管预测(STP)方法,通过几何先验提升LLM的数据效率,实验证明其在NL-RX-SYNTH数据集上以16倍更少数据达到基线准确率。

摘要:

大型语言模型(LLMs)遵循一致的扩展定律——经验幂律拟合,预测损失如何随着计算、数据和参数减少。虽然具有预测性,但这些定律是描述性而非指导性的:它们描述了典型的训练,而不是最优的训练。令人惊讶的是,很少有工作成功挑战了这些定律所暗示的数据效率界限——这是我们主要关注的重点。为此,我们引入了测地假设,认为令牌序列在平滑的语义流形上追踪测地线,因此是局部线性的。基于这一原理,我们提出了一种新的语义管预测(STP)任务,一种类似于JEPA的正则化方法,将隐藏状态轨迹限制在测地线的管状邻域内。STP将JEPA推广到语言中,而无需显式多视图增强。我们显示这种约束提高了信噪比,并因此通过防止推理过程中的轨迹碰撞来保持多样性。实证上,STP使LLMs在NL-RX-SYNTH数据集上以16倍更少的训练数据达到基线准确率,直接违反了Chinchilla风格的扩展定律,并证明了有原则的几何先验可以超越暴力扩展。代码可在https://github.com/galilai-group/llm-jepa#stp获取。

英文摘要:

Large Language Models (LLMs) obey consistent scaling laws -- empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these laws are descriptive rather than prescriptive: they characterize typical training, not optimal training. Surprisingly few works have successfully challenged the data-efficiency bounds implied by these laws -- which is our primary focus. To that end, we introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear. Building on this principle, we propose a novel Semantic Tube Prediction (STP) task, a JEPA-style regularizer that confines hidden-state trajectories to a tubular neighborhood of the geodesic. STP generalizes JEPA to language without requiring explicit multi-view augmentations. We show this constraint improves signal-to-noise ratio, and consequently preserves diversity by preventing trajectory collisions during inference. Empirically, STP allows LLMs to match baseline accuracy with 16$\times$ less training data on the NL-RX-SYNTH dataset, directly violating the data term of Chinchilla-style scaling laws and demonstrating that principled geometric priors can surpass brute-force scaling. Code is available at https://github.com/galilai-group/llm-jepa#stp.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑