发表机构
Stanford University; University of California, Santa Barbara（斯坦福大学; 加利福尼亚大学圣巴巴拉分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文证明词共现统计的群不变性决定嵌入几何，将圆与品客流形推广至任意有限群、紧致群及齐次空间，并在月份、和弦及天体表示中验证了不可约表示结构。
AI 中文摘要
语言模型为月份、星期和地点等概念学习到的表示展现出一致的几何结构：圆形和马鞍形的“品客”流形。近期工作将这些结构追溯到词共现统计中的平移对称性，并在共现仅依赖于阿贝尔概念格上距离时推导出所观察到的傅里叶几何。我们证明更一般的对称性概念同样能导致结构化的预测。考虑由任意有限群、紧致群和齐次空间定义的对称性，我们证明：只要一个词族的共现统计在群G下不变，学习到的词嵌入就由G的不可约表示（irreps）的矩阵元构成。当G为循环群时，圆和品客形状出现，此时不可约表示即为傅里叶模式。我们在三个实验设置中验证了不可约表示结构。(i) 循环群Z₁₂：对于一年中的月份，我们重现了已知的圆形几何。(ii) 作用于大三和弦与小三和弦的二面体群：我们统一了两个经典观察——移调与和弦转位构成一个作用于和弦的群（T/I）（音乐理论），这蕴含了著名的“五度圈”在学习到的和弦嵌入中涌现（机器学习）。(iii) 我们解释并重现了近期发现的大语言模型（LLMs）中天体的球面表示，将其作为由我们理论导出的球谐嵌入。我们的结果表明，学习到的表示的几何往往是底层数据统计对称性的结果。
英文摘要
The representations that language models learn for concepts such as months, weekdays, and places display consistent geometric structure: circles and saddle-shaped "Pringle" manifolds. Recent work traced these structures to $\textit{translation symmetry}$ in word co-occurrence statistics, deriving the observed Fourier geometry when co-occurrence depends only on distance on an abelian lattice of concepts. We demonstrate that more general notions of symmetry lead to equally structured predictions. Considering symmetries defined by arbitrary finite groups, compact groups, and homogeneous spaces, we prove that whenever the co-occurrence statistics of a word family are invariant under a group $G$, the learned word embeddings consist of matrix elements of the irreducible representations (irreps) of $G$. Circles and Pringles arise when $G$ is cyclic, in which case the irreps are Fourier modes. We verify the irrep structure in three experimental settings. (i) The cyclic group $\mathbb{Z}_{12}$: for the months of the year we recover the known circular geometry. (ii) A dihedral group acting on the major and minor triads: we unify two classical observations -- that transposition and chord inversion form a group ($T/I$) acting on chords (music theory), which $\textit{implies}$ that the well-known "circle of fifths" emerges in learned chord embeddings (machine learning). (iii) We explain and reproduce a recently discovered spherical representation of celestial objects in large language models (LLMs) as a spherical-harmonic embedding derived from our theory. Our results demonstrate that the geometry of learned representations is often a consequence of the statistical symmetry of underlying data.
Comments31 pages, 9 figures
