arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.02765cs.LGcs.AI
收藏

表示能力:Transformer语言模型中特征表示的几何限制

Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models

Arizona State University(亚利桑那州立大学)

Alexander Guha

AI总结:

基于线性表示和叠加假设,通过嵌入矩阵的余弦相似度分布估计模型可支持的近正交方向数量,推导出容量公式,并发现容量对偏差ε指数敏感。

摘要:

模型维度($d_{model}$)是Transformer语言模型中的一个基本超参数,但其在设定特征表示的几何限制方面的作用仍未得到充分探索。基于线性表示和叠加假设——这些假设提出模型将特征编码为潜在空间中的近正交方向——我们开发了一个框架来估计模型可以支持多少个这样的方向。我们首先将嵌入矩阵确立为跨潜在空间近正交约束的可测量代理:成对余弦相似度分布中有意义的token关系与偶然相似性之间的边界给出了模型对完美正交性的可接受偏差ε的具体估计。将此度量应用于数十个开源模型揭示了两个类别:具有高ε且其嵌入缺乏近正交结构的模型,以及具有低ε且保持近正交结构的模型。然后我们表明,标准的Johnson-Lindenstrauss引理大大低估了训练表示的填充效率,并推导出一个调整后的容量公式,其中近正交方向的数量取决于向量与维度的比率($k/d$)而非原始计数——这一单一修改在没有额外参数的情况下将预测误差降低了两个数量级。结合这些结果,我们将表示能力定义为模型潜在空间中可用于特征和嵌入的可区分方向上界。容量对ε指数敏感,并且较大的模型倾向于更严格的正交约束而非最大化原始容量——这一模式与几种解释(稳定性-容量权衡、可用概念的上限或模型规模的混杂因素)兼容,我们将这些留给未来工作。

英文摘要:

Model dimension ($d_{model}$) is a fundamental hyperparameter in transformer language models, yet its role in setting the geometric limits of feature representation remains under-explored. Grounded in the Linear Representation and Superposition Hypotheses - which propose that models encode features as near-orthogonal directions in latent space - we develop a framework for estimating how many such directions a model can support. We first establish the embedding matrix as a measurable proxy for near-orthogonality constraints across the latent space: the boundary between meaningful token relationships and incidental similarity in the pairwise cosine similarity distribution gives a concrete estimate of the model's accepted deviation $\varepsilon$ from perfect orthogonality. Applying this metric across dozens of open-source models reveals two classes: models with high $\varepsilon$ whose embeddings lack near-orthogonal structure, and models with low $\varepsilon$ that maintain it. We then show that the standard Johnson-Lindenstrauss lemma greatly underestimates the packing efficiency of trained representations, and derive an adjusted capacity formula in which the number of near-orthogonal directions depends on the ratio of vectors to dimensions ($k/d$) rather than the raw count - a single modification that cuts prediction error by two orders of magnitude with no extra parameters. Combining these results, we define representational capacity as an upper bound on the number of distinguishable directions available for features and embeddings in a model's latent space. Capacity is exponentially sensitive to $\varepsilon$, and larger models favor tighter orthogonality constraints over maximizing raw capacity - a pattern compatible with several explanations (a stability-capacity trade-off, a ceiling on usable concepts, or confounds with model scale) that we leave to future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑