来自词嵌入的新世界：大语言模型中小世界词汇语义网络的比较研究
From the New World of Word Embeddings: A Comparative Study of Small-World Lexico-Semantic Networks in LLMs
AI总结：
本研究构建不同参数规模LLMs输入嵌入的词汇语义网络，发现其具有小世界特性，且更大规模LLMs的网络更复杂、小世界效应更弱，还通过多类分析验证了方法有效性。
摘要：
词汇语义网络将单词表示为节点，将其语义相关性表示为边。传统上这类网络使用基于编码器的模型的嵌入或静态向量构建，而仅解码器大语言模型（LLMs）的嵌入仍未得到充分研究。与编码器模型不同，LLMs以预测下一个词元为目标进行训练，不会直接编码当前词元的含义。本文构建了不同参数规模LLMs输入嵌入的词汇语义网络，并对其全局和局部结构进行了比较分析。结果表明，这些网络表现出小世界特性，具有高聚类性和短路径长度的特征。此外，规模更大的LLMs会产生更复杂的网络，小世界效应更弱、路径更长，反映出更丰富的语义结构和关系。我们通过对常见概念对、源自WordNet的结构化词汇关系以及定性词的跨语言语义网络的分析，进一步验证了该方法。
英文摘要：
Lexico-semantic networks represent words as nodes and their semantic relatedness as edges. While such networks are traditionally constructed using embeddings from encoder-based models or static vectors, embeddings from decoder-only large language models (LLMs) remain underexplored. Unlike encoder models, LLMs are trained with a next-token prediction objective, which does not directly encode the meaning of the current token. In this paper, we construct lexico-semantic networks from the input embeddings of LLMs with varying parameter scales and conduct a comparative analysis of their global and local structures. Our results show that these networks exhibit small-world properties, characterized by high clustering and short path lengths. Moreover, larger LLMs yield more intricate networks with less small-world effects and longer paths, reflecting richer semantic structures and relations. We further validate our approach through analyses of common conceptual pairs, structured lexical relations derived from WordNet, and a cross-lingual semantic network for qualitative words.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
