The complicated syntax structure of natural language is hard to be explicitly modeled by sequence-based models. Graph is a natural structure to describe the complicated relation between tokens. The recent advance in Graph Neural Networks (GNN) provides a powerful tool to model graph structure data, but simple graph models such as Graph Convolutional Networks (GCN) suffer from over-smoothing problem, that is, when stacking multiple layers, all nodes will converge to the same value. In this paper, we propose a novel Recursive Graphical Neural Networks model (ReGNN) to represent text organized in the form of graph. In our proposed model, LSTM is used to dynamically decide which part of the aggregated neighbor information should be transmitted to upper layers thus alleviating the over-smoothing problem. Furthermore, to encourage the exchange between the local and global information, a global graph-level node is designed. We conduct experiments on both single and multiple label text classification tasks. Experiment results show that our ReGNN model surpasses the strong baselines significantly in most of the datasets and greatly alleviates the over-smoothing problem.
Byte-Level Recursive Convolutional Auto-Encoder for Text
Courant Institute of Mathematical Sciences, New York University(纽约大学库朗数学科学研究所);Facebook AI Research, Facebook Inc.(脸书人工智能研究院,脸书公司);Center for Data Science, New York University(纽约大学数据科学中心)
University of Amsterdam(阿姆斯特丹大学);Microsoft Research(微软研究院)
近年来,自动文本摘要取得了长足的进步。随着这一进展,出现了一个问题,即通常由自动摘要模型生成的摘要类型是否符合用户的需求。Ter Hoeve et al(2020)否定地回答了这个问题。除其他外,他们建议专注于生成具有更多图形元素的摘要。这与我们从心理语言学文献中了解到的人类如何处理文本是一致的。从这两个角度出发,我们提出了一个新的任务:总结与图形元素,我们验证了这些摘要是有帮助的临界质量的人。我们收集了一个高质量的人类标记数据集,以支持对任务的研究。我们提出了一些基线方法,表明任务是有趣的和具有挑战性的。因此,通过这项工作,我们希望在自动摘要社区中激发一条新的研究路线。
Institute for Natural Language Processing(自然语言处理研究所);University of Stuttgart(斯图加特大学);Center for Information and Language Processing(信息与语言处理中心);University of Munich(慕尼黑大学)