Are pairs of words that tend to occur together also likely to stand in a linguistic dependency? This empirical question is motivated by a long history of literature in cognitive science, psycholinguistics, and NLP. In this work we contribute an extensive analysis of the relationship between linguistic dependencies and statistical dependence between words. Improving on previous work, we introduce the use of large pretrained language models to compute contextualized estimates of the pointwise mutual information between words (CPMI). For multiple models and languages, we extract dependency trees which maximize CPMI, and compare to gold standard linguistic dependencies. Overall, we find that CPMI dependencies achieve an unlabelled undirected attachment score of at most $\approx 0.5$. While far above chance, and consistently above a non-contextualized PMI baseline, this score is generally comparable to a simple baseline formed by connecting adjacent words. We analyze which kinds of linguistic dependencies are best captured in CPMI dependencies, and also find marked differences between the estimates of the large pretrained language models, illustrating how their different training schemes affect the type of dependencies they capture.
We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained …
Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models
Center for Information and Language Processing (CIS)（信息与语言处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；ScaDS.AI and TU Dresden（ScaDS.AI 和 梅克伦堡-前波美拉尼亚技术大学）；Department of Statistics, LMU Munich（统计学系，慕尼黑大学）；University of Maryland, College Park（马里兰大学 College Park 分校）
Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?
ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）；Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
As the name implies, contextualized representations of language are typically motivated by their ability to encode context. Which aspects of context are captured by such representations? We introduce an approach to address this question using Representational Similarity Analysis (RSA). As case studies, we investigate the degree to which a verb embedding encodes the verb's subject, a pronoun embedding encodes the pronoun's antecedent, and a full-sentence representation encodes the sentence's head word (as determined by a dependency parse). In all cases, we show that BERT's contextualized embedd…
TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models
ATILF, University of Lorraine - CNRS, France（ATILF，洛林大学 - CNRS，法国）；Center for Data Science in Humanities, Chosun University, South Korea（人文数据科学中心，全州大学，韩国）
Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations
What if Deception Cannot be Detected? A Cross-Linguistic Study on the Limits of Deception Detection from Text
Institut für Maschinelle Sprachverarbeitung, University of Stuttgart, Germany（机器语言处理研究所，斯图加特大学，德国）；Fundamentals of Natural Language Processing, University of Bamberg, Germany（自然语言处理基础， Bamberg大学，德国）；Leibniz Institute for Psychology, Trier, Germany（莱布尼茨心理学研究所，特里尔，德国）
Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
Uppsala University（乌普萨拉大学）；The Ohio State University（俄亥俄州立大学）；University of Turku（图尔库大学）；Charles University in Prague（布拉格查理大学）；Stanford University（斯坦福大学）；Indiana University（印第安纳大学）
Socioeconomic Dependencies of Linguistic Patterns in Twitter: A Multivariate Analysis
Univ Lyon（里昂大学）；ENS de Lyon（里昂高等师范学院）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；UCB Lyon 1（里昂第一大学）；LIP UMR 5668（里昂信息处理实验室（UMR 5668））；IXXI（IXXI（实验室））；ICAR UMR 5191（ICAR实验室（UMR 5191））；Lidilem（Lidilem（机构））；University of Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；u-grenoble3（格勒诺布尔第三大学）
K-UD: Revising Korean Universal Dependencies Guidelines
Seoul National University（首尔大学）；The Chinese University of Hong Kong（香港中文大学）；The University of British Columbia（不列颠哥伦比亚大学）；Sorbonne Université（索邦大学）；SeoulTech（首尔科学技术大学）；Anyang University（安阳大学）
On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation
Dublin City University（都柏林城市大学）；School of Computing（计算机学院）
新的机器翻译（MT）技术正在迅速出现，并且伴随着它们，实现人类平等的大胆主张，例如：（i）产生的结果接近“平均双语人类翻译者所实现的准确性”（Wu等人，2017 b）或（ii）“与专业翻译人员相比，翻译质量与人类同等”（Hassan等人，2018）已经看到了曙光（Laubli et al.，2018年）。除了这些论文中的许多人自己对人类平等的定义之外，这些耸人听闻的说法往往没有得到对翻译所涉及的所有方面的完整分析的支持。确定机器翻译统计方法的优势与人类翻译方式之间的差异是我们研究的出发点。通过观察机器翻译输出和语言学理论，我们能够确定一些剩余的问题。这些问题从简单的数字和性别一致错误到更复杂的现象，如体值和时态的正确翻译。我们的实验证实，与其他研究一起（Bentivogli等人，2016年），神经MT在许多方面已经超过了统计MT。然而，一些问题仍然存在，另一些问题已经出现。我们涵盖了一系列与将特定语言特征整合到统计和神经MT中相关的问题，旨在分析其中一些问题并提供解决方案。我们的工作重点是解决围绕语言学和机器翻译之间的复杂关系的三个主要研究问题。我们识别出了自动翻译系统所缺乏的语言信息，以便产生更准确的翻译，并将额外的功能集成到现有的管道中。我们确定过度泛化或“算法偏见”作为神经MT的潜在缺点，并将其与许多剩余的语言问题联系起来。
Manual evaluation is essential to judge progress on automatic text summarization. However, we conduct a survey on recent summarization system papers that reveals little agreement on how to perform such evaluation studies. We conduct two evaluation experiments on two aspects of summaries' linguistic quality (coherence and repetitiveness) to compare Likert-type and ranking annotations and show that best choice of evaluation method can vary from one aspect to another. In our survey, we also find that study parameters such as the overall number of annotators and distribution of annotators to annot…
Beijing Language and Culture University（北京语言大学）；Anhui Jianzhu University（安徽建筑大学）；Zhejiang University（浙江大学）；Guangdong University of Foreign Studies（广东外语外贸大学）
Traditional linguistic theories have largely regard language as a formal system composed of rigid rules. However, their failures in processing real language, the recent successes in statistical natural language processing, and the findings of many psychological experiments have suggested that language may be more a probabilistic system than a formal system, and thus cannot be faithfully modeled with the either/or rules of formal linguistic theory. The present study, based on authentic language data, confirmed that those important linguistic issues, such as linguistic universal, diachronic drif…
A New Statistical Parser Based on Bigram Lexical Dependencies
本文描述了一种新的统计分析器，它是基于概率的依赖关系的中心词在分析树。标准的bigram概率估计技术扩展到计算单词对之间的依赖关系的概率。使用《华尔街日报》数据的测试表明，该方法的性能至少与SPATTER（Magerman 95，Jelinek et al 94）一样好，后者在此任务上具有统计分析器的最佳结果。该方法的简单性意味着该模型在15分钟内训练了40，000个句子。使用波束搜索策略，解析速度可以提高到每分钟200多个句子，准确性损失可以忽略不计。
The Chinese University of Hong Kong（香港中文大学）；The University of British Columbia（不列颠哥伦比亚大学）；Hanbat National University（韩巴国立大学）；TeddySum；Indiana University（印第安纳大学）
In this study, we propose a morpheme-based scheme for Korean dependency parsing and adopt the proposed scheme to Universal Dependencies. We present the linguistic rationale that illustrates the motivation and the necessity of adopting the morpheme-based format, and develop scripts that convert between the original format used by Universal Dependencies and the proposed morpheme-based format automatically. The effectiveness of the proposed format for Korean dependency parsing is then testified by both statistical and neural models, including UDPipe and Stanza, with our carefully constructed morp…
Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；Key Laboratory of Shanghai Education Commission for Intelligent Interactionand Cognitive Engineering, Shanghai Jiao Tong University（上海市教育委员会智能交互与认知工程重点实验室（上海交通大学））；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（教育部人工智能重点实验室（上海交通大学人工智能研究院））
Both syntactic and semantic structures are key linguistic contextual clues, in which parsing the latter has been well shown beneficial from parsing the former. However, few works ever made an attempt to let semantic parsing help syntactic parsing. As linguistic representation formalisms, both syntax and semantics may be represented in either span (constituent/phrase) or dependency, on both of which joint learning was also seldom explored. In this paper, we propose a novel joint model of syntactic and semantic parsing on both span and dependency representations, which incorporates syntactic inf…
Leibniz-Zentrum Allgemeine Sprachwissenschaft（莱布尼茨普通语言学中心）；University of Edinburgh（爱丁堡大学）
In this paper, we launch a new Universal Dependencies treebank for an endangered language from Amazonia: Kakataibo, a Panoan language spoken in Peru. We first discuss the collaborative methodology implemented, which proved effective to create a treebank in the context of a Computational Linguistic course for undergraduates. Then, we describe the general details of the treebank and the language-specific considerations implemented for the proposed annotation. We finally conduct some experiments on part-of-speech tagging and syntactic dependency parsing. We focus on monolingual and transfer learn…
University of Amsterdam（阿姆斯特丹大学）；University of Milano-Bicocca（米兰比可卡大学）；Facebook AI Research（Facebook人工智能研究院）；Catalan Institute for Research and Advanced Studies（加泰罗尼亚研究与高级研究所）；Universitat Pompeu Fabra（庞培法布拉大学）；College de France（法兰西学院）
Recursive processing in sentence comprehension is considered a hallmark of human linguistic abilities. However, its underlying neural mechanisms remain largely unknown. We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing, namely the storing of grammatical number and gender information in working memory and its use in long-distance agreement (e.g., capturing the correct number agreement between subject and verb when they are separated by other phrases). Although the network, a recurrent architecture with…
University School for Advanced Studies IUSS Pavia（帕维亚高等研究院大学）；Ecole Polytechnique Federale de Lausanne（洛桑联邦理工学院）；University of Geneva（日内瓦大学）；Yale University（耶鲁大学）
A sharp tension exists about the nature of human language between two opposite parties: those who believe that statistical surface distributions, in particular using measures like surprisal, provide a better understanding of language processing, vs. those who believe that discrete hierarchical structures implementing linguistic information such as syntactic ones are a better tool. In this paper, we show that this dichotomy is a false one. Relying on the fact that statistical measures can be defined on the basis of either structural or non-structural models, we provide empirical evidence that o…
Stockholm University（斯德哥尔摩大学）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）
The patterns in which the syntax of different languages converges and diverges are often used to inform work on cross-lingual transfer. Nevertheless, little empirical work has been done on quantifying the prevalence of different syntactic divergences across language pairs. We propose a framework for extracting divergence patterns for any language pair from a parallel corpus, building on Universal Dependencies. We show that our framework provides a detailed picture of cross-language divergences, generalizes previous approaches, and lends itself to full automation. We further present a novel dat…
