首页/论文/2012.00857

arXiv 2012.008572020-12cs.CLcs.AIcs.LG

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling

Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Google Research（谷歌研究院）

Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, Aaron Courville

备注：: Published as a conference paper at ACL 2021

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one or several corresponded words. While previous unsupervised parsing methods mostly focus on only inducing one class of grammars, we introduce a novel model, StructFormer, that can simultaneously induce dependency and constituency structure. To achieve this, we propose a new parsing framework that can jointly generate a constituency tree and dependency graph. Then we integrate the induced dependency relations into the transformer, in a differentiable manner, through a novel dependency-constrained self-attention mechanism. Experimental results show that our model can achieve strong results on unsupervised constituency parsing, unsupervised dependency parsing, and masked language modeling at the same time.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Dependency Induction Through the Lens of Visual Perception

Electrical and Computer Engineering, Carnegie Mellon University（卡内基梅隆大学电气与计算机工程系）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）

Most previous work on grammar induction focuses on learning phrasal or dependency structure purely from text. However, because the signal provided by text alone is limited, recently introduced visually grounded syntax models make use of multimodal information leading to improved performance in constituency grammar induction. However, as compared to dependency grammars, constituency grammars do not provide a straightforward way to incorporate visual information without enforcing language-specific heuristics. In this paper, we propose an unsupervised grammar induction model that leverages word c…

2021-09-20cs.CL

Improved Latent Tree Induction with Distant Supervision via Span Constraints

University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）

For over thirty years, researchers have developed and analyzed methods for latent tree induction as an approach for unsupervised syntactic parsing. Nonetheless, modern systems still do not perform well enough compared to their supervised counterparts to have any practical use as structural annotation of text. In this work, we present a technique that uses distant supervision in the form of span constraints (i.e. phrase bracketing) to improve performance in unsupervised constituency parsing. Using a relatively small number of span constraints we can substantially improve the output from DIORA, …

2021-11-01cs.CL

Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders

University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）

We introduce deep inside-outside recursive autoencoders (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree. Our approach predicts each word in an input sentence conditioned on the rest of the sentence and uses inside-outside dynamic programming to consider all possible binary trees over the sentence. At test time the CKY algorithm extracts the highest scoring parse. DIORA achieves a new state-of-the-art F1 in unsupervised binary constituency parsing (unlabeled) in two benchmark datasets, WSJ and MultiN…

2019-04-04cs.CL

基于轻度上下文有关文法的无监督不连续成分句法分析

Unsupervised Discontinuous Constituency Parsing with Mildly Context-Sensitive Grammars

ShanghaiTech University（上海科技大学）；MIT（麻省理工学院）

该研究针对无监督不连续成分句法分析问题，采用LCFRS-2文法并优化规则降低推理复杂度，结合张量分解与嵌入参数化扩展非终结符数量，在德荷语上可归纳出有语言学意义的句法树。

2023-06-09cs.CL

Grammar Induction with Neural Language Models: An Unusual Replication

New York University（纽约大学）

A substantial thread of recent work on latent tree learning has attempted to develop neural network models with parse-valued latent variables and train them on non-parsing tasks, in the hope of having them discover interpretable tree structure. In a recent paper, Shen et al. (2018) introduce such a model and report near-state-of-the-art results on the target task of language modeling, and the first strong latent tree learning result on constituency parsing. In an attempt to reproduce these results, we discover issues that make the original results hard to trust, including tuning and even train…

2018-08-29cs.CL

Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

Seoul National University（首尔国立大学）；University of Chicago（芝加哥大学）

With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in corre…

2020-01-30cs.AIcs.CLcs.LG

更多相关论文收起更多论文

从语言模型中归纳语言结构

Linguistic Structure Induction from Language Models

INSTITUT FÜR INFORMATIK（信息学研究所）

本论文研究从语言模型中无监督地归纳成分和依存句法结构，重点分析StructFormer模型，通过六个实验评估其性能，并参与BabyLM挑战赛，结果表明改造transformer模型以诱导句法结构具有可行性。

2024-03-11cs.AIcs.CL

通过隐式结构构建提升受认知启发的数据高效语言模型的性能

Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building

Heinrich Heine University（海因里希·海涅大学）

本研究针对数据高效语言模型预训练问题，采用融入层级结构偏置的StructFormer及其变体模型参与BabyLM 2023挑战赛，在39项任务中部分任务表现优于普通Transformer，但未全面超越RoBERTa基线。

2023-10-31cs.CL

利用语法归纳实现语言理解与生成

Leveraging Grammar Induction for Language Understanding and Generation

Shanghai Jiao Tong University（上海交通大学）

本研究提出一种无监督语法归纳方法，将归纳的语法特征作为句法掩码融入Transformer引导自注意力，在机器翻译与自然语言理解任务上性能优于基线模型，且适用于从零训练和预训练场景。

2024-10-07cs.AIcs.CL

Video-aided Unsupervised Grammar Induction

University of Rochester（罗切斯特大学）；Tencent AI Lab（腾讯AI实验室）

We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from video…

2021-05-04cs.CLcs.CV

Depth-bounding is effective: Improvements and evaluation of unsupervised PCFG induction

The Ohio State University（俄亥俄州立大学）；Harvard University（哈佛大学）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

There have been several recent attempts to improve the accuracy of grammar induction systems by bounding the recursive complexity of the induction model (Ponvert et al., 2011; Noji and Johnson, 2016; Shain et al., 2016; Jin et al., 2018). Modern depth-bounded grammar inducers have been shown to be more accurate than early unbounded PCFG inducers, but this technique has never been compared against unbounded induction within the same system, in part because most previous depth-bounding models are built around sequence models, the complexity of which grows exponentially with the maximum allowed d…

2018-09-10cs.CL

Unsupervised Recurrent Neural Network Grammars

Harvard University（哈佛大学）；University of Oxford（牛津大学）；DeepMind（深度思维）

Recurrent neural network grammars (RNNG) are generative models of language which jointly model syntax and surface structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. Supervised RNNGs achieve strong language modeling and parsing performance, but require an annotated corpus of parse trees. In this work, we experiment with unsupervised learning of RNNGs. Since directly marginalizing over the space of latent trees is intractable, we instead apply amortized variational inference. To maximize the evidence lower bound, we develop an inference network p…

2019-08-05cs.CLstat.ML

Visually Grounded Compound PCFGs

University of Edinburgh（爱丁堡大学）；University of Amsterdam（阿姆斯特丹大学）

Exploiting visual groundings for language understanding has recently been drawing much attention. In this work, we study visually grounded grammar induction and learn a constituency parser from both unlabeled text and its visual groundings. Existing work on this task (Shi et al., 2019) optimizes a parser via Reinforce and derives the learning signal only from the alignment of images and sentences. While their model is relatively accurate overall, its error distribution is very uneven, with low performance on certain constituents types (e.g., 26.2% recall on verb phrases, VPs) and high on other…

2020-09-25cs.CLcs.CV

Deep Clustering of Text Representations for Supervision-free Probing of Syntax

ShareChat；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；ETH Zurich（苏黎世联邦理工学院）

We explore deep clustering of text representations for unsupervised model interpretation and induction of syntax. As these representations are high-dimensional, out-of-the-box methods like KMeans do not work well. Thus, our approach jointly transforms the representations into a lower-dimensional cluster-friendly space and clusters them. We consider two notions of syntax: Part of speech Induction (POSI) and constituency labelling (CoLab) in this work. Interestingly, we find that Multilingual BERT (mBERT) contains surprising amount of syntactic knowledge of English; possibly even as much as Engl…

2021-12-01cs.CL