Large pre-trained language models (LMs) have demonstrated impressive capabilities in generating long, fluent text; however, there is little to no analysis on their ability to maintain entity coherence and consistency. In this work, we focus on the end task of narrative generation and systematically analyse the long-range entity coherence and consistency in generated stories. First, we propose a set of automatic metrics for measuring model performance in terms of entity usage. Given these metrics, we quantify the limitations of current LMs. Next, we propose augmenting a pre-trained LM with a dynamic entity memory in an end-to-end manner by using an auxiliary entity-related loss for guiding the reads and writes to the memory. We demonstrate that the dynamic entity memory increases entity coherence according to both automatic and human judgment and helps preserving entity-related information especially in settings with a limited context window. Finally, we also validate that our automatic metrics are correlated with human ratings and serve as a good indicator of the quality of generated stories.
Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives?
Department of Computer Science and Engineering, University of Minnesota（计算机科学与工程系，明尼苏达大学）；Department of Educational Psychology, University of Minnesota（教育心理学系，明尼苏达大学）；Department of Linguistics, Hamline University（语言学系，哈姆林大学）；Department of English, University of Minnesota（英语系，明尼苏达大学）
Narrative Analysis of True Crime Podcasts With Knowledge Graph-Augmented Large Language Models
Carleton College（卡尔顿学院）；Claremont McKenna College（克莱蒙特麦肯纳学院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Loyola Marymount University（洛约拉马利蒙特大学）；California Institute of Technology（加州理工学院）
Curation and Extraction of Drug-Related Entities from Reddit Platform
Population Health Sciences, Weill Cornell Medicine, New York City, USA（威立·科恩医学中心人口健康科学系，纽约市，美国）；School of Computing and Information Systems, University of Melbourne, Melbourne, Australia（墨尔本大学计算机与信息系统学院，墨尔本，澳大利亚）；Emergency Medicine, Weill Cornell Medicine, New York City, USA（急诊医学，威立·科恩医学中心，纽约市，美国）
Research Training Group AIPHES（AIPHES研究培训小组）；Heidelberg University（海德堡大学）
Despite recent successes of large pre-trained language models in solving reasoning tasks, their inference capabilities remain opaque. We posit that such models can be made more interpretable by explicitly generating interim inference rules, and using them to guide the generation of task-specific textual outputs. In this paper we present COINS, a recursive inference framework that i) iteratively reads context sentences, ii) dynamically generates contextualized inference rules, encodes them, and iii) uses them to guide task-specific output generation. We apply COINS to a Narrative Story Completi…
Tsinghua University（清华大学）；Guangdong OPPO Mobile Telecommunications Corp., Ltd.（广东OPPO移动通信有限公司）；NetEase Inc.（网易公司）
Despite advances in generating fluent texts, existing pretraining models tend to attach incoherent event sequences to involved entities when generating narratives such as stories and news. We conjecture that such issues result from representing entities as static embeddings of superficial words, while neglecting to model their ever-changing states, i.e., the information they carry, as the text unfolds. Therefore, we extend the Transformer model to dynamically conduct entity state updates and sentence realization for narrative generation. We propose a contrastive framework to learn the state re…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University at Buffalo（布法罗大学）；NVIDIA（英伟达）
Relations between entities can be represented by different instances, e.g., a sentence containing both entities or a fact in a Knowledge Graph (KG). However, these instances may not well capture the general relations between entities, may be difficult to understand by humans, even may not be found due to the incompleteness of the knowledge source. In this paper, we introduce the Open Relation Modeling problem - given two entities, generate a coherent sentence describing the relation between them. To solve this problem, we propose to teach machines to generate definition-like relation descripti…
The Challenge and Reward of Fair Play in Narrative: A Computational Approach
Department of Computer Science Hebrew University of Jerusalem（计算机科学系 Hebrew University of Jerusalem）；Department of Law and Digital Humanities Hebrew University of Jerusalem（法律与数字人文系 Hebrew University of Jerusalem）
NARRA-Gym for Evaluating Interactive Narrative Agents
University of Notre Dame（诺丁汉大学）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Pennsylvania（宾夕法尼亚大学）；Lehigh University（莱恩大学）；Massachusetts Institute of Technology（麻省理工学院）；Bake AI UC Santa Barbara（加州大学圣芭芭拉分校）；University of Washington（华盛顿大学）
Understanding narrative text requires capturing characters' motivations, goals, and mental states. This paper proposes an Entity-based Narrative Graph (ENG) to model the internal-states of characters in a story. We explicitly model entities, their interactions and the context in which they appear, and learn rich representations for them. We experiment with different task-adaptive pre-training objectives, in-domain training, and symbolic inference to capture dependencies between different decisions in the output space. We evaluate our model on two narrative understanding tasks: predicting chara…
We propose a method for controlled narrative/story generation where we are able to guide the model to produce coherent narratives with user-specified target endings by interpolation: for example, we are told that Jim went hiking and at the end Jim needed to be rescued, and we want the model to incrementally generate steps along the way. The core of our method is an interpolation model based on GPT-2 which conditions on a previous sentence and a next sentence in a narrative and fills in the gap. Additionally, a reranker helps control for coherence of the generated text. With human evaluation, w…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Human understanding of narrative texts requires making commonsense inferences beyond what is stated explicitly in the text. A recent model, COMET, can generate such implicit commonsense inferences along several dimensions such as pre- and post-conditions, motivations, and mental states of the participants. However, COMET was trained on commonsense inferences of short phrases, and is therefore discourse-agnostic. When presented with each sentence of a multi-sentence narrative, it might generate inferences that are inconsistent with the rest of the narrative. We present the task of discourse-awa…
IRC-Bench: Recognizing Entities from Contextual Cues in First-Person Reminiscences
Intelligent Systems, Afeka Academic College of Engineering（阿法卡学术工程学院智能系统）；School of Computer Science, Faculty of Sciences, Holon Institute of Technology（霍隆理工学院计算机科学学院）
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent Inc.（腾讯公司）
Procedural text understanding requires machines to reason about entity states within the dynamical narratives. Current procedural text understanding approaches are commonly \textbf{entity-wise}, which separately track each entity and independently predict different states of each entity. Such an entity-wise paradigm does not consider the interaction between entities and their states. In this paper, we propose a new \textbf{scene-wise} paradigm for procedural text understanding, which jointly tracks states of all entities in a scene-by-scene manner. Based on this paradigm, we propose \textbf{S}…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Georgia Institute of Technology（佐治亚理工学院）；University of California San Diego（加州大学圣地亚哥分校）
Set expansion aims to expand a small set of seed entities into a complete set of relevant entities. Most existing approaches assume the input seed set is unambiguous and completely ignore the multi-faceted semantics of seed entities. As a result, given the seed set {"Canon", "Sony", "Nikon"}, previous models return one mixed set of entities that are either Camera Brands or Japanese Companies. In this paper, we study the task of multi-faceted set expansion, which aims to capture all semantic facets in the seed set and return multiple sets of entities, one for each semantic facet. We propose an …
A Counterfactual Graph Approach to Oppositional Narrative Analysis
Faculty of Engineering, University of Deusto, Bilbao, Spain（德乌斯托大学工程学院，比斯开，西班牙）；Deusto Institute of Technology, University of Deusto, Bilbao, Spain（德乌斯托技术研究所，德乌斯托大学，比斯开，西班牙）
MBZUAI, UAE（阿联酋马斯喀特大学）；Nazarbayev University, Kazakhstan（哈萨克斯坦纳扎尔巴耶夫大学）；University of Maryland, USA（美国马里兰大学）；University of Arizona, USA（美国亚利桑那大学）；Athens University of Economics and Business, Greece（希腊雅典经济与商业大学）；University of Porto, Portugal（葡萄牙波尔图大学）；Sofia University "St. Kliment Ohridski", Bulgaria（保加利亚索菲亚大学）；University of Helsinki, Finland（芬兰赫尔辛基大学）；Beira Interior and INESC TEC, Portugal（葡萄牙贝拉内尔和INESC TEC）；Porto and INESC TEC, Portugal（葡萄牙波尔图和INESC TEC）；University of Padova, Italy（意大利帕多瓦大学）；European Commission Joint Research Center, Italy（意大利欧洲委员会联合研究中心）；Polish Academy of Sciences, Poland（波兰科学院）
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
We propose the task of outline-conditioned story generation: given an outline as a set of phrases that describe key characters and events to appear in a story, the task is to generate a coherent narrative that is consistent with the provided outline. This task is challenging as the input only provides a rough sketch of the plot, and thus, models need to generate a story by interweaving the key points provided in the outline. This requires the model to keep track of the dynamic states of the latent plot, conditioning on the input outline while generating the full story. We present PlotMachines,…
In this work, we investigate the human perception of coherence in open-domain dialogues. In particular, we address the problem of annotating and modeling the coherence of next-turn candidates while considering the entire history of the dialogue. First, we create the Switchboard Coherence (SWBD-Coh) corpus, a dataset of human-human spoken dialogues annotated with turn coherence ratings, where next-turn candidate utterances ratings are provided considering the full dialogue context. Our statistical analysis of the corpus indicates how turn coherence perception is affected by patterns of distribu…
Conflicts, Villains, Resolutions: Towards models of Narrative Media Framing
The University of Melbourne（墨尔本大学）；The Hong Kong Polytechnic University（香港理工大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61）；The Australian National University（澳大利亚国立大学）
Stony Brook University（石溪大学）；United States Naval Academy（美国海军学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Early work on narrative modeling used explicit plans and goals to generate stories, but the language generation itself was restricted and inflexible. Modern methods use language models for more robust generation, but often lack an explicit representation of the scaffolding and dynamics that guide a coherent narrative. This paper introduces a new model that integrates explicit narrative structure with neural language models, formalizing narrative modeling as a Switching Linear Dynamical System (SLDS). A SLDS is a dynamical system in which the latent dynamics of the system (i.e. how the state ve…
Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks. When a long summary must be produced to appropriately cover the facets of that text, that summary needs to present a coherent narrative to be understandable by a reader, but current automatic and human evaluation methods fail to identify gaps in coherence. In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for …
Narrative story generation is a challenging problem because it demands the generated sentences with tight semantic connections, which has not been well studied by most existing generative models. To address this problem, we propose a skeleton-based model to promote the coherence of generated stories. Different from traditional models that generate a complete sentence at a stroke, the proposed model first generates the most critical phrases, called skeleton, and then expands the skeleton to a complete and fluent sentence. The skeleton is not manually defined, but learned by a reinforcement lear…
STORYTELLER: An Enhanced Plot-Planning Framework for Coherent and Cohesive Story Generation
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；UNSW Sydney（新南威尔士大学）；Chongqing University（重庆大学）
Hefei Institutes of Physical Science, Chinese Academy of Sciences（合肥物理科学研究院，中国科学院）；University of Science and Technology of China（中国科学技术大学）；Changzhou University（常州大学）
Automated storytelling has long captured the attention of researchers for the ubiquity of narratives in everyday life. However, it is challenging to maintain coherence and stay on-topic toward a specific ending when generating narratives with neural language models. In this paper, we introduce Story generation with Reader Models (StoRM), a framework in which a reader model is used to reason about the story should progress. A reader model infers what a human reader believes about the concepts, entities, and relations about the fictional story world. We show how an explicit reader model represen…
