In this work, we want to give an overview on which pragmatic abilities have been tested in LLMs so far and how these tests have been carried out. To do this, we first discuss the scope of the field of pragmatics and suggest a subdivision into discourse pragmatics and interactional pragmatics. We give a non-exhaustive overview of the phenomena of those two subdomains and the methods traditionally used to analyze them. We subsequently consider the resulting heterogeneous set of phenomena and methods as a starting point for our survey of work on discourse pragmatics and interactional pragmatics in the context of LLMs.
Michigan State University（密歇根州立大学）；University of Chicago（芝加哥大学）
As pre-trained language models (LMs) continue to dominate NLP, it is increasingly important that we understand the depth of language capabilities in these models. In this paper, we target pre-trained LMs' competence in pragmatics, with a focus on pragmatics relating to discourse connectives. We formulate cloze-style tests using a combination of naturally-occurring data and controlled inputs drawn from psycholinguistics. We focus on testing models' ability to use pragmatic cues to predict discourse connectives, models' ability to understand implicatures relating to connectives, and the extent t…
The Pragmatic Persona: Discovering LLM Persona through Bridging Inference
Department of Artificial Intelligence, Chung-Ang University, Seoul, 06974, Republic of Korea（Chung-Ang大学人工智能系）；Department of Computer Science, University of British Columbia, Vancouver, BC V6T 1Z4, Canada（不列颠哥伦比亚大学计算机科学系）；Van Lang University, Ho Chi Minh City, Vietnam（文-lang大学）
Research in the social sciences and psychology has shown that the persuasiveness of an argument depends not only the language employed, but also on attributes of the source/communicator, the audience, and the appropriateness and strength of the argument's claims given the pragmatic and discourse context of the argument. Among these characteristics of persuasive arguments, prior work in NLP does not explicitly investigate the effect of the pragmatic and discourse context when determining argument quality. This paper presents a new dataset to initiate the study of this aspect of argumentation: i…
On Theoretically-Driven LLM Agents for Multi-Dimensional Discourse Analysis
Laboratory of The New Ethos, Warsaw University of Technology, Poland（新伦理实验室，华沙理工大学，波兰）；Faculty of Electronics and Information Technology, Warsaw University of Technology, Poland（电子与信息技术学院，华沙理工大学，波兰）
An account of utterance interpretation in discourse needs to face the issue of how the discourse context controls the space of interacting preferences. Assuming a discourse processing architecture that distinguishes the grammar and pragmatics subsystems in terms of monotonic and nonmonotonic inferences, I will discuss how independently motivated default preferences interact in the interpretation of intersentential pronominal anaphora. In the framework of a general discourse processing model that integrates both the grammar and pragmatics subsystems, I will propose a fine structure of the prefe…
The Impact of Synthetic Data Augmentation on Discourse-Pragmatic Function Classification
Institute of Linguistics（语言学研究所）；College of Liberal Arts and Sciences, University of Florida（佛罗里达大学文理学院）；Faculty of Arts and Humanities, Heinrich Heine University Düsseldorf（杜塞尔多夫海因里希·海涅大学艺术与人文学院）；Sun Yat-sen University（中山大学）
The University of Melbourne（墨尔本大学）；AI Lab at ByteDance（字节跳动AI实验室）
While pretrained language models ("LM") have driven impressive gains over morpho-syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better understanding of their discourse modelling capabilities, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ documents constructed from English Wikipedia and CNN news articles. O…
Today's artificial assistants are typically prompted to perform tasks through direct, imperative commands such as \emph{Set a timer} or \emph{Pick up the box}. However, to progress toward more natural exchanges between humans and these assistants, it is important to understand the way non-imperative utterances can indirectly elicit action of an addressee. In this paper, we investigate command types in the setting of a grounded, collaborative game. We focus on a less understood family of utterances for eliciting agent action, locatives like \emph{The chair is in the other room}, and demonstrate…
In order to enrich dynamic semantic theories with a `pragmatic' capacity, we combine dynamic and nonmonotonic (preferential) logics in a modal logic setting. We extend a fragment of Van Benthem and De Rijke's dynamic modal logic with additional preferential operators in the underlying static logic, which enables us to define defeasible (pragmatic) entailments over a given piece of discourse. We will show how this setting can be used for a dynamic logical analysis of preferential resolutions of ambiguous pronouns in discourse.
In this paper we present ongoing work to produce an expressive TTS reader that can be used both in text and dialogue applications. The system called SPARSAR has been used to read (English) poetry so far but it can now be applied to any text. The text is fully analyzed both at phonetic and phonological level, and at syntactic and semantic level. In addition, the system has access to a restricted list of typical pragmatically marked phrases and expressions that are used to convey specific discourse function and speech acts and need specialized intonational contours. The text is transformed into …
In the quest to give a formal compositional semantics to natural languages, semanticists have started turning their attention to phenomena that have been also considered as parts of pragmatics (e.g., discourse anaphora and presupposition projection). To account for these phenomena, the very kinds of meanings assigned to words and phrases are often revisited. To be more specific, in the prevalent paradigm of modeling natural language denotations using the simply-typed lambda calculus (higher-order logic) this means revisiting the types of denotations assigned to individual parts of speech. Howe…
Information Sciences Institute（信息科学研究所）；University of Southern California（南加州大学）
Storytelling, whether via fables, news reports, documentaries, or memoirs, can be thought of as the communication of interesting and related events that, taken together, form a concrete process. It is desirable to extract the event chains that represent such processes. However, this extraction remains a challenging problem. We posit that this is due to the nature of the texts from which chains are discovered. Natural language text interleaves a narrative of concrete, salient events with background information, contextualization, opinion, and other elements that are important for a variety of n…
Learning to Diagnose and Correct Moral Errors: Beyond Shallow Heuristics in Moral Alignment
University of Mississippi（密西根州立大学）；Nanyang Technological University（南洋理工大学）；Northeastern University（东北大学）；AWS AI Labs（AWS AI实验室）；Johns Hopkins University（约翰霍普金斯大学）；Qualcomm（高通）；Michigan State University（密西根州立大学）；Indiana University Indianapolis（印第安纳大学印第安纳波利斯分校）
Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
University of Washington（华盛顿大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Samaya AI（Samaya人工智能）；Mistral AI（Mistral人工智能）；Allen Institute for AI（人工智能研究所）
