深层语言表示中可分离流形的出现
Emergence of Separable Manifolds in Deep Language Representations
摘要：
深度神经网络（DNN）在解决各种认知模式的感知任务方面取得了很大的经验成功。虽然它们只是松散地受到生物大脑的启发，但最近的研究报告了从任务优化的DNN和大脑中的神经种群中提取的表示之间的相当大的相似性。DNN随后成为一种流行的模型类，用于推断复杂认知功能背后的计算原理，反过来，它们也成为应用最初开发用于探测神经群体信息的方法的自然测试平台。在这项工作中，我们利用平均场理论流形分析，从计算神经科学，连接几何特征表示与线性可分性的类，分析语言表示从大规模的上下文嵌入模型的最新技术。我们探索不同模型家族（BERT、RoBERTa、GPT等）的表示。并找到跨层深度的语言流形出现的证据（例如，词性标记的流形），尤其是在模糊数据（即，具有多个词性标记的单词，或者包括许多单词的词性类）中。此外，我们发现这些流形中线性可分性的出现是由流形的半径、维数和流形间相关性的联合减少所驱动的。
英文摘要：
Deep neural networks (DNNs) have shown much empirical success in solving perceptual tasks across various cognitive modalities. While they are only loosely inspired by the biological brain, recent studies report considerable similarities between representations extracted from task-optimized DNNs and neural populations in the brain. DNNs have subsequently become a popular model class to infer computational principles underlying complex cognitive functions, and in turn, they have also emerged as a natural testbed for applying methods originally developed to probe information in neural populations. In this work, we utilize mean-field theoretic manifold analysis, a recent technique from computational neuroscience that connects geometry of feature representations with linear separability of classes, to analyze language representations from large-scale contextual embedding models. We explore representations from different model families (BERT, RoBERTa, GPT, etc.) and find evidence for emergence of linguistic manifolds across layer depth (e.g., manifolds for part-of-speech tags), especially in ambiguous data (i.e, words with multiple part-of-speech tags, or part-of-speech classes including many words). In addition, we find that the emergence of linear separability in these manifolds is driven by a combined reduction of manifolds' radius, dimensionality and inter-manifold correlations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
