我们身边的文本内容每天都在增长。就在我们说话的同时，在线报纸、博客或社交媒体上也有大量文章被撰写出来。同样，AI领域的最新进展（如语言模型或传统经典AI方法）正在利用上述所有内容来改进其学习到的表示，从而以接近人类的准确率应对NLP挑战。人们普遍认为，要应对文本摘要、问答、机器翻译乃至代词消解等挑战，获取来自有效来源的高质量文本至关重要。例如，要做好摘要，需要选择最重要的句子，并将它们拼接起来形成摘要。然而，如果我们无法获得格式良好的英语句子，甚至是无效句子，会发生什么？尽管获得高质量句子非常重要，但找出验证这些句子的方法仍是一个开放的研究领域。为解决这一问题，我们提出了一种简化的英语句子验证方式，即通过一种新颖的神经符号方法。近来，神经符号方法在应对各种NLP挑战方面引发了越来越多的关注，因为它们正证明自身作为各类AI系统核心组成部分的有效性。通过将经典AI与现代AI相结合，即将语法和句法规则与语言模型相融合，我们有效应对了Corpus of Linguistic Acceptability（COLA，语言可接受性语料库）任务，该任务用于判断一个词序列是否为符合英语语法的句子。除其他结果外，所开展的实验有效表明，将符号系统与非符号系统相融合，有助于前者为后者的准确率结果提供洞察。
英文摘要：
Textual content around us is growing on a daily basis. Numerous articles are being written as we speak on online newspapers, blogs, or social media. Similarly, recent advances in the AI field, like language models or traditional classic AI approaches, are utilizing all the above to improve their learned representation to tackle NLP challenges with human-like accuracy. It is commonly accepted that it is crucial to have access to well-written text from valid sources to tackle challenges like text summarization, question-answering, machine translation, or even pronoun resolution. For instance, to summarize well, one needs to select the most important sentences in order to concatenate them to form the summary. However, what happens if we do not have access to well-formed English sentences or even non-valid sentences? Despite the importance of having access to well-written sentences, figuring out ways to validate them is still an open area of research. To address this problem, we present a simplified way to validate English sentences through a novel neural-symbolic approach. Lately, neural-symbolic approaches have triggered an increasing interest towards tackling various NLP challenges, as they are demonstrating their effectiveness as a central component in various AI systems. Through combining Classic with Modern AI, which involves the blending of grammatical and syntactical rules with language models, we effectively tackle the Corpus of Linguistic Acceptability (COLA), a task that shows whether or not a sequence of words is an English grammatical sentence. Among others, undertaken experiments effectively show that blending symbolic and non-symbolic systems helps the former provide insights about the latter's accuracy results.
The ALTA Institute（ALTA研究所）；Computer Laboratory（计算机实验室）；University of Cambridge（剑桥大学）
Shortage of available training data is holding back progress in the area of automated error detection. This paper investigates two alternative methods for artificially generating writing errors, in order to create additional resources. We propose treating error generation as a machine translation task, where grammatically correct text is translated to contain errors. In addition, we explore a system for extracting textual patterns from an annotated corpus, which can then be used to insert errors into grammatically correct sentences. Our experiments show that the inclusion of artificially gener…
Language models align with human judgments on key grammatical constructions
Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University（哈佛大学坎普ner自然与人工智能研究所）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Georgia Tech（佐治亚理工学院）；Massachusetts Institute of Technology（麻省理工学院）
Recent works in Grammatical Error Correction (GEC) have leveraged the progress in Neural Machine Translation (NMT), to learn rewrites from parallel corpora of grammatically incorrect and corrected sentences, achieving state-of-the-art results. At the same time, Generative Adversarial Networks (GANs) have been successful in generating realistic texts across many different tasks by learning to directly minimize the difference between human-generated and synthetic text. In this work, we present an adversarial learning approach to GEC, using the generator-discriminator framework. The generator is …
A First Context-Free Grammar Applied to Nawatl Corpora Augmentation
Laboratoire Informatique d’Avignon（阿维尼翁信息实验室）；Avignon Université（阿维尼昂大学）；Facultad de Matemáticas（数学系）；Instituto de Investigaciones en Educación（教育研究所）；Universidad Veracruzana（韦拉克鲁斯大学）
Sri Lanka Institute of Information Technology（斯里兰卡信息技术学院）；Murdoch University（莫道克大学）
Machine Translation (MT) is an area in natural language processing, which focus on translating from one language to another. Many approaches ranging from statistical methods to deep learning approaches are used in order to achieve MT. However, these methods either require a large number of data or a clear understanding about the language. Sinhala language has less digital text which could be used to train a deep neural network. Furthermore, Sinhala has complex rules therefore, it is harder to create statistical rules in order to apply statistical methods in MT. This research focuses on Sinhala…
Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncoder
Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Idiap Research Institute（Idiap研究机构）；National Biomarker Centre, CRUK-MI, Univ. of Manchester（国家生物标记中心，CRUK-MI，曼彻斯特大学）
Neural network based approaches to automated story plot generation attempt to learn how to generate novel plots from a corpus of natural language plot summaries. Prior work has shown that a semantic abstraction of sentences called events improves neural plot generation and and allows one to decompose the problem into: (1) the generation of a sequence of events (event-to-event) and (2) the transformation of these events into natural language sentences (event-to-sentence). However, typical neural language generation approaches to event-to-sentence can ignore the event details and produce grammat…
The University of Hong Kong（香港大学）；Noah’s Ark Lab, Huawei Technologies（华为技术诺亚方舟实验室）
In this paper, we propose phraseNet, a neural machine translator with a phrase memory which stores phrase pairs in symbolic form, mined from corpus or specified by human experts. For any given source sentence, phraseNet scans the phrase memory to determine the candidate phrase pairs and integrates tagging information in the representation of source sentence accordingly. The decoder utilizes a mixture of word-generating component and phrase-generating component, with a specifically designed strategy to generate a sequence of multiple words all at once. The phraseNet not only approaches one step…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；IIIS, Tsinghua University（清华大学交叉信息研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Deepmind（深度思考）；MIT BCS（麻省理工学院脑与认知科学系）；CBMM（脑、心智与机器中心）
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns…
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
We present Referee, a novel framework for sentence summarization that can be trained reference-free (i.e., requiring no gold summaries for supervision), while allowing direct control for compression ratio. Our work is the first to demonstrate that reference-free, controlled sentence summarization is feasible via the conceptual framework of Symbolic Knowledge Distillation (West et al., 2022), where latent knowledge in pre-trained language models is distilled via explicit examples sampled from the teacher models, further purified with three types of filters: length, fidelity, and Information Bot…
A symbolic Perl algorithm for the unification of Nahuatl word spellings
ICTT, Avignon Université（阿维尼翁大学ICTT）；Inst. de Investigaciones en Educación, Universidad Veracruzana（韦拉克鲁斯大学教育研究所）；GIL, Universidad Nacional Autónoma de México（墨西哥国立自治大学GIL）；Avignon, France（法国阿维尼翁）
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language. We factorize PIGLeT into a physical dynamics model, and a separate language model. Our dynamics model learns not just what objects are but also what they do: glass cups break when thrown, plastic ones don't. We then use it as the interface to our language model, giving us a unified model of linguistic form and grounded meaning. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that result through a literal symbolic …
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
Linking human whole-body motion and natural language is of great interest for the generation of semantic representations of observed human behaviors as well as for the generation of robot behaviors based on natural language input. While there has been a large body of research in this area, most approaches that exist today require a symbolic representation of motions (e.g. in the form of motion primitives), which have to be defined a-priori or require complex segmentation algorithms. In contrast, recent advances in the field of neural networks and especially deep learning have demonstrated that…
We implement a divide-and-concur iterative projection approach to context-free grammar inference. Unlike most state-of-the-art models of natural language processing, our method requires a relatively small number of discrete parameters, making the inferred grammar directly interpretable -- one can read off from a solution how to construct grammatically valid sentences. Another advantage of our approach is the ability to infer meaningful grammatical rules from just a few sentences, compared to the hundreds of gigabytes of training data many other models employ. We demonstrate several ways of app…
Measuring vagueness and subjectivity in texts: from symbolic to neural VAGO
Institut Jean-Nicod（让·尼科研究所）；CNRS（法国国家科学研究中心）；ENS-PSL（巴黎高等师范学院-巴黎文理研究大学）；EHESS（法国高等社会科学研究院）；IRISA-CNRS（法国国家信息与自动化研究所-法国国家科学研究中心）；Direction Générale de l’Armement（法国军备总局）；Mondeca（蒙代卡公司）
Johns Hopkins University（约翰斯·霍普金斯大学）；Citadel（Citadel公司）；University of Florida（佛罗里达大学）；Microsoft Research AI（微软人工智能研究院）
CAPTCHAs based on reading text are susceptible to machine-learning-based attacks due to recent significant advances in deep learning (DL). To address this, this paper promotes image/visual captioning based CAPTCHAs, which is robust against machine-learning-based attacks. To develop image/visual-captioning-based CAPTCHAs, this paper proposes a new image captioning architecture by exploiting tensor product representations (TPR), a structured neural-symbolic framework developed in cognitive science over the past 20 years, with the aim of integrating DL with explicit language structures and rules.…
Deep neural networks based on layer-stacking architectures have historically suffered from poor inherent interpretability. Meanwhile, symbolic probabilistic models function with clear interpretability, but how to combine them with neural networks to enhance their performance remains to be explored. In this paper, we try to marry these two systems for text classification via a structured language model. We propose a Symbolic-Neural model that can learn to explicitly predict class labels of text spans from a constituency tree without requiring any access to span-level gold labels. As the structu…
MIT（麻省理工学院）；IBM Research（IBM研究院）；University of Oregon（俄勒冈大学）
Entity linking (EL), the task of disambiguating mentions in text by linking them to entities in a knowledge graph, is crucial for text understanding, question answering or conversational systems. Entity linking on short text (e.g., single sentence or question) poses particular challenges due to limited context. While prior approaches use either heuristics or black-box neural methods, here we propose LNN-EL, a neuro-symbolic approach that combines the advantages of using interpretable rules based on first-order logic with the performance of neural learning. Even though constrained to using rule…
Plymouth University（普利茅斯大学）；Bernstein Center for Computational Neuroscience Berlin（柏林伯恩斯坦计算神经科学中心）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Inria Sophia-Antipolis Méditerranée（法国国家信息与自动化研究所索菲亚-安蒂波利斯地中海研究中心）
Computation is classically studied in terms of automata, formal languages and algorithms; yet, the relation between neural dynamics and symbolic representations and operations is still unclear in traditional eliminative connectionism. Therefore, we suggest a unique perspective on this central issue, to which we would like to refer as to transparent connectionism, by proposing accounts of how symbolic computation can be implemented in neural substrates. In this study we first introduce a new model of dynamics on a symbolic space, the versatile shift, showing that it supports the real-time simul…
Utrecht Institute of Linguistics OTS, Utrecht University（乌得勒支大学乌得勒支语言学研究所OTS）；LIRMM, Université de Montpellier, CNRS（法国国家科学研究中心蒙彼利埃大学计算机信息与机器人微电子实验室）
Linear logic and the linear λ-calculus have a long standing tradition in the study of natural language form and meaning. Among the proof calculi of linear logic, proof nets are of particular interest, offering an attractive geometric representation of derivations that is unburdened by the bureaucratic complications of conventional prooftheoretic formats. Building on recent advances in set-theoretic learning, we propose a neural variant of proof nets based on Sinkhorn networks, which allows us to translate parsing as the problem of extracting syntactic primitives and permuting them into alignme…
