We develop the information-theoretical concepts required to study the statistical dependencies among three variables. Some of such dependencies are pure triple interactions, in the sense that they cannot be explained in terms of a combination of pairwise correlations. We derive bounds for triple dependencies, and characterize the shape of the joint probability distribution of three binary variables with high triple interaction. The analysis also allows us to quantify the amount of redundancy in the mutual information between pairs of variables, and to assess whether the information between two variables is or is not mediated by a third variable. These concepts are applied to the analysis of written texts. We find that the probability that a given word is found in a particular location within the text is not only modulated by the presence or absence of other nearby words, but also, on the presence or absence of nearby pairs of words. We identify the words enclosing the key semantic concepts of the text, the triplets of words with high pairwise and triple interactions, and the words that mediate the pairwise interactions between other words.
A New Statistical Parser Based on Bigram Lexical Dependencies
本文描述了一种新的统计分析器,它是基于概率的依赖关系的中心词在分析树。标准的bigram概率估计技术扩展到计算单词对之间的依赖关系的概率。使用《华尔街日报》数据的测试表明,该方法的性能至少与SPATTER(Magerman 95,Jelinek et al 94)一样好,后者在此任务上具有统计分析器的最佳结果。该方法的简单性意味着该模型在15分钟内训练了40,000个句子。使用波束搜索策略,解析速度可以提高到每分钟200多个句子,准确性损失可以忽略不计。
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Stanford University(斯坦福大学)
我们研究了如何以无监督的方式掩蔽和预测令牌可以产生语言结构和下游性能增益。最近的理论表明,预训练的语言模型通过隐含地充当下游任务的完形填空缩减的掩码来获得有用的归纳偏差。虽然有吸引力,我们表明,在实践中使用的随机掩蔽策略的成功,不能解释这样的完形填空式的面具。我们使用三个不同分类数据集的特定任务词典构建了类似完形填空的掩码,并表明大多数预训练的性能增益来自与词典无关的通用掩码。为了解释这些通用面具的经验成功,我们证明了掩蔽语言模型(MLM)的目标和现有的方法之间的对应关系,学习统计依赖的图形模型。使用此,我们推导出一种方法,用于提取这些学习的统计依赖MLMs,并表明这些依赖编码有用的归纳偏见的形式的句法结构。在无监督解析评估中,简单地在隐含的统计依赖结构上形成最小生成树优于无监督解析的经典方法(58.74 vs. 55.91 UUAS)。
Towards the quantification of the semantic information encoded in written language
The University of Manchester(曼彻斯特大学);Consejo Nacional de Investigaciones Científicas y Técnicas(阿根廷国家科学技术研究委员会);Centro Atómico Bariloche(巴里洛切原子中心);Instituto Balseiro(巴尔塞罗研究所)
Escuela Politécnica Superior, Universidad Autónoma de Madrid(马德里自治大学高等理工学院);UNED(西班牙国立远程教育大学);The Abdus Salam International Center for Theoretical Physics(阿卜杜斯·萨拉姆国际理论物理中心)
Labelled network subgraphs reveal stylistic subtleties in written texts
Institute of Mathematics and Computer Science, University of São Paulo(圣保罗大学数学与计算机科学研究所);Department of Computer Science, University of Toronto(多伦多大学计算机科学系)