Meaning Representations (AMRs) are broad-coverage sentence-level semantic graphs. Existing approaches to generating text from AMR have focused on training sequence-to-sequence or graph-to-sequence models on AMR annotated data only. In this paper, we propose an alternative approach that combines a strong pre-trained language model with cycle consistency-based re-scoring. Despite the simplicity of the approach, our experimental results show these models outperform all previous techniques on the English LDC2017T10dataset, including the recent use of transformer architectures. In addition to the standard evaluation metrics, we provide human evaluation experiments that further substantiate the strength of our approach.
Institute of Computer Science（计算机科学研究所）；University of Tartu（塔尔图大学）
This study explores enhancing grammatical error correction (GEC) through artificial error generation (AEG) using language models (LMs). Specifically, we fine-tune Llama 2-based LMs for error generation and find that this approach yields synthetic errors akin to human errors. Next, we train GEC Llama models with the help of these artificial errors and outperform previous state-of-the-art error correction models, with gains ranging between 0.8 and 6 F0.5 points across all tested languages (German, Ukrainian, and Estonian). Moreover, we demonstrate that generating errors by fine-tuning smaller se…
We introduce ASQ, a tool to automatically mine questions and answers from a sentence using the Abstract Meaning Representation (AMR). Previous work has used question-answer pairs to specify the predicate-argument structure of a sentence using natural language, which does not require linguistic expertise or training, and created datasets such as QA-SRL and QAMR, for which the question-answer pair annotations were crowdsourced. Our goal is to build a tool (ASQ) that maps from the traditional meaning representation AMR to a question-answer meaning representation (QMR). This enables construction o…
Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning
Strong AI Lab, NAOInstitute, Waipapa Taumata Rau - The University of Auckland（强人工智能实验室、NAO研究所、怀帕帕·陶马陶·劳 - 奥克兰大学）；Xtracta, New Zealand（新西兰Xtracta公司）；University of Cambridge, UK（英国剑桥大学）；Sun Yat-Sen University, China（中国中山大学）；The University of Auckland, New Zealand（新西兰奥克兰大学）
Meaning Representation (AMR; Banarescu et al., 2013) encodes the meaning of sentences as a directed graph and Smatch (Cai and Knight, 2013) is the primary metric for evaluating AMR graphs. Smatch, however, is unaware of some meaning-equivalent variations in graph structure allowed by the AMR Specification and gives different scores for AMRs exhibiting these variations. In this paper I propose four normalization methods for helping to ensure that conceptually equivalent AMRs are evaluated as equivalent. Equivalent AMRs with and without normalization can look quite different---comparing a gold c…
Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today
Tsinghua University（清华大学）；The First Affiliated Hospital of Nanchang University（南昌大学第一附属医院）；University of Chinese Academy of Sciences（中国科学院大学）；Shandong University（山东大学）；East China Normal University（华东师范大学）
该研究在两个真实临床数据集上比较 GPT-4 与传统 AI 工具及医生的痴呆诊断表现，发现 GPT-4 虽具潜力，但准确性、可解释性和忠实性目前仍未超越传统工具。
This demonstration paper presents StreamSide, an open-source toolkit for annotating multiple kinds of meaning representations. StreamSide supports frame-based annotation schemes e.g., Abstract Meaning Representation (AMR) and frameless annotation schemes e.g., Widely Interpretable Semantic Representation (WISeR). Moreover, it supports both sentence-level and document-level annotation by allowing annotators to create multi-rooted graphs for input text. It can open and automatically convert between several types of input formats including plain text, Penman notation, and its own JSON format enab…
Singapore University of Technology and Design（新加坡科技设计大学）
Novel concepts are essential for design innovation and can be generated with the aid of data stimuli and computers. However, current generative design algorithms focus on diagrammatic or spatial concepts that are either too abstract to understand or too detailed for early phase design exploration. This paper explores the uses of generative pre-trained transformers (GPT) for natural language design concept generation. Our experiments involve the use of GPT-2 and GPT-3 for different creative reasonings in design tasks. Both show reasonably good performance for verbal design concept generation.
Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）
We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vigne…
Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4
McGill University（麦吉尔大学）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Eindhoven University of Technology（埃因霍温理工大学）；University of Pennsylvania（宾夕法尼亚大学）；European University Institute（欧洲大学研究院）
Knowledge base question answering (KBQA)is an important task in Natural Language Processing. Existing approaches face significant challenges including complex question understanding, necessity for reasoning, and lack of large end-to-end training datasets. In this work, we propose Neuro-Symbolic Question Answering (NSQA), a modular KBQA system, that leverages (1) Abstract Meaning Representation (AMR) parses for task-independent question understanding; (2) a simple yet effective graph transformation approach to convert AMR parses into candidate logical queries that are aligned to the KB; (3) a p…
Tilburg University（蒂尔堡大学）；University College London（伦敦大学学院）
Language models such as GPT-3 have caused a furore in the research community. Some studies found that GPT-3 has some creative abilities and makes mistakes that are on par with human behaviour. This paper answers a related question: Who is GPT-3? We administered two validated measurement tools to GPT-3 to assess its personality, the values it holds and its self-reported demographics. Our results show that GPT-3 scores similarly to human samples in terms of personality and - when provided with a model response memory - in terms of the values it holds. We provide the first evidence of psychologic…
Thomas Jefferson University（托马斯杰斐逊大学）；Weil-Cornell Medical Center（威尔康奈尔医学中心）；The Ohio State University（俄亥俄州立大学）；The Jackson Lab（杰克逊实验室）；Unity Health（联合健康）
Purpose: The purpose of this study was to determine if an ensemble of multiple LLM agents could be used collectively to provide a more reliable assessment of a pixel-based AI triage tool than a single LLM. Methods: 29,766 non-contrast CT head exams from fourteen hospitals were processed by a commercial intracranial hemorrhage (ICH) AI detection tool. Radiology reports were analyzed by an ensemble of eight open-source LLM models and a HIPAA compliant internal version of GPT-4o using a single multi-shot prompt that assessed for presence of ICH. 1,726 examples were manually reviewed. Performance …
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Tencent AI Lab（腾讯AI实验室）；The Chinese University of Hong Kong（香港中文大学）
Evaluating Text Summaries Generated by Large Language Models Using OpenAI's GPT
University of Colorado - Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；University of Kansas（堪萨斯大学）；DataCrunch Lab, LLC（DataCrunch实验室有限责任公司）；University of Florida（佛罗里达大学）
