In this paper, we introduce an important yet relatively unexplored NLP task called Multi-Narrative Semantic Overlap (MNSO), which entails generating a Semantic Overlap of multiple alternate narratives. As no benchmark dataset is readily available for this task, we created one by crawling 2,925 narrative pairs from the web and then, went through the tedious process of manually creating 411 different ground-truth semantic overlaps by engaging human annotators. As a way to evaluate this novel task, we first conducted a systematic study by borrowing the popular ROUGE metric from text-summarization literature and discovered that ROUGE is not suitable for our task. Subsequently, we conducted further human annotations/validations to create 200 document-level and 1,518 sentence-level ground-truth labels which helped us formulate a new precision-recall style evaluation metric, called SEM-F1 (semantic F1). Experimental results show that the proposed SEM-F1 metric yields higher correlation with human judgement as well as higher inter-rater-agreement compared to ROUGE metric.
Narrative is a ubiquitous component of human communication. Understanding its structure plays a critical role in a wide variety of applications, ranging from simple comparative analyses to enhanced narrative retrieval, comprehension, or reasoning capabilities. Prior research in narratology has highlighted the importance of studying the links between cognitive and linguistic aspects of narratives for effective comprehension. This interdependence is related to the textual semantics and mental language in narratives, referring to characters' motivations, feelings or emotions, and beliefs. However…
While large language models have shown exciting progress on several NLP benchmarks, evaluating their ability for complex analogical reasoning remains under-explored. Here, we introduce a high-quality crowdsourced dataset of narratives for employing proverbs in context as a benchmark for abstract language understanding. The dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and contains minimal lexical overlaps between narratives and proverbs, ensuring that models need to go beyond surface-level reasoning to succeed. We explore three tasks: (1) proverb re…
From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation
Department of Computer Science, University of Hamburg（汉堡大学计算机科学系）；Department of Socioeconomics, University of Hamburg（汉堡大学社会经济系）；AI and Explanability Group, Leuphana University of Lüneburg（吕内堡大学人工智能与可解释性小组）
Amazon（亚马逊）；Bloomberg（彭博）；University of Groningen（格罗宁根大学）；University of Amsterdam（阿姆斯特丹大学）；Ahold Delhaize Research（阿霍德德尔海兹研究院）
Writers such as journalists often use automatic tools to find relevant content to include in their narratives. In this paper, we focus on supporting writers in the news domain to develop event-centric narratives. Given an incomplete narrative that specifies a main event and a context, we aim to retrieve news articles that discuss relevant events that would enable the continuation of the narrative. We formally define this task and propose a retrieval dataset construction procedure that relies on existing news articles to simulate incomplete narratives and relevant articles. Experiments on two d…
Department of Computer Science, Purdue University（普渡大学计算机科学系）
We examine a new commonsense reasoning task: given a narrative describing a social interaction that centers on two protagonists, systems make inferences about the underlying relationship trajectory. Specifically, we propose two evaluation tasks: Relationship Outlook Prediction MCQ and Resolution Prediction MCQ. In Relationship Outlook Prediction, a system maps an interaction to a relationship outlook that captures how the interaction is expected to change the relationship. In Resolution Prediction, a system attributes a given relationship outlook to a particular resolution that explains the ou…
Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA)（达莱莫勒人工智能研究所）
We present two deep learning approaches to narrative text understanding for character relationship modelling. The temporal evolution of these relations is described by dynamic word embeddings, that are designed to learn semantic changes over time. An empirical analysis of the corresponding character trajectories shows that such approaches are effective in depicting dynamic evolution. A supervised learning approach based on the state-of-the-art transformer model BERT is used instead to detect static relations between characters. The empirical validation shows that such events (e.g., two charact…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Human understanding of narrative texts requires making commonsense inferences beyond what is stated explicitly in the text. A recent model, COMET, can generate such implicit commonsense inferences along several dimensions such as pre- and post-conditions, motivations, and mental states of the participants. However, COMET was trained on commonsense inferences of short phrases, and is therefore discourse-agnostic. When presented with each sentence of a multi-sentence narrative, it might generate inferences that are inconsistent with the rest of the narrative. We present the task of discourse-awa…
The University of Melbourne（墨尔本大学）；School of Computing and Information Systems（计算与信息系统学院）
Story comprehension requires a deep semantic understanding of the narrative, making it a challenging task. Inspired by previous studies on ROC Story Cloze Test, we propose a novel method, tracking various semantic aspects with external neural memory chains while encouraging each to focus on a particular semantic aspect. Evaluated on the task of story ending prediction, our model demonstrates superior performance to a collection of competitive baselines, setting a new state of the art.
The Chinese University of Hong Kong（香港中文大学）；Microsoft Research, Redmond（微软研究院（雷德蒙德））；Center for Theoretical Physics, Massachusetts Institute of Technology（麻省理工学院理论物理中心）
We propose the task of narrative incoherence detection as a new arena for inter-sentential semantic understanding: Given a multi-sentence narrative, decide whether there exist any semantic discrepancies in the narrative flow. Specifically, we focus on the missing sentence and discordant sentence detection. Despite its simple setup, this task is challenging as the model needs to understand and analyze a multi-sentence narrative, and predict incoherence at the sentence level. As an initial step towards this task, we implement several baselines either directly analyzing the raw text (\textit{toke…
The University of Surrey（萨里大学）；The University of Sheffield（谢菲尔德大学）
Story generation aims to generate a long narrative conditioned on a given input. In spite of the success of prior works with the application of pre-trained models, current neural models for Chinese stories still struggle to generate high-quality long text narratives. We hypothesise that this stems from ambiguity in syntactically parsing the Chinese language, which does not have explicit delimiters for word segmentation. Consequently, neural models suffer from the inefficient capturing of features in Chinese narratives. In this paper, we present a new generation framework that enhances the feat…
In this work, we propose to use linguistic annotations as a basis for a \textit{Discourse-Aware Semantic Self-Attention} encoder that we employ for reading comprehension on long narrative texts. We extract relations between discourse units, events and their arguments as well as coreferring mentions, using available annotation tools. Our empirical evaluation shows that the investigated structures improve the overall performance, especially intra-sentential and cross-sentential discourse relations, sentence-internal semantic role relations, and long-distance coreference relations. We show that d…
University of California Santa Cruz（加州大学圣克鲁兹分校）；Columbia University（哥伦比亚大学）
In order to tell stories in different voices for different audiences, interactive story systems require: (1) a semantic representation of story structure, and (2) the ability to automatically generate story and dialogue from this semantic representation using some form of Natural Language Generation (NLG). However, there has been limited research on methods for linking story structures to narrative descriptions of scenes and story events. In this paper we present an automatic method for converting from Scheherazade's story intention graph, a semantic representation, to the input required by th…
Temporal Narrative Monitoring in Dynamic Information Environments
School of Information Science（信息科学学院）；University of Washington（华盛顿大学）；Whiting School of Engineering（工程学院）；Johns Hopkins University（约翰霍普金斯大学）；School of Computer Science（计算机科学学院）；Carnegie Mellon University（卡内基梅隆大学）；Human Centered Design Engineering（以人为核心的设计工程）
LLMs for automatic annotation of Mandarin narrative transcripts
Shanghai Jiao Tong University（上海交通大学）；University of California San Diego（加州大学圣地亚哥分校）；The Hong Kong Polytechnic University（香港理工大学）；National Research Center for Language and Well-Being（语言与幸福感国家研究中心）；City University of Hong Kong（香港城市大学）
Multi-Level Narrative Evaluation Outperforms Lexical Features for Mental Health
Institute for Artificial Intelligence（人工智能研究院）；School of Psychological and Cognitive Sciences（心理学与认知科学学院）；School of Intelligence Science and Technology（智能科学与技术学院）；State Key Laboratory of General AI（通用人工智能国家重点实验室）；Beijing Key Laboratory of Behavior and Mental Health（北京行为与心理健康重点实验室）；PKU-Changsha Institute for Computing and Digital Economy（北京大学长沙计算与数字经济研究院）
Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generation
Tehran Institute for Advanced Studies, Khatam University, Iran（德黑兰高级研究所，卡塔姆大学，伊朗）；School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran（电气与计算机工程学院，工程学院，德黑兰大学，伊朗）
Detecting Narrative Shifts through Persistent Structures: A Topological Analysis of Media Discourse
Department of Cyber Intelligence and Data Science（网络智能与数据科学系）；Biological and Computational Intelligence Center（生物与计算智能中心）；National Intelligence University（国家智能大学）
In the era of clinical information explosion, a good strategy for clinical text summarization is helpful to improve the clinical workflow. The ideal summarization strategy can preserve important information in the informative but less organized, ill-structured clinical narrative texts. Instead of using pure statistical learning approaches, which are difficult to interpret and explain, we utilized knowledge of computational linguistics with human experts-curated biomedical knowledge base to achieve the interpretable and meaningful clinical text summarization. Our research objective is to use th…
This paper describes the Duluth systems that participated in Task 14 of SemEval 2016, Semantic Taxonomy Enrichment. There were three related systems in the formal evaluation which are discussed here, along with numerous post--evaluation runs. All of these systems identified synonyms between WordNet and other dictionaries by measuring the gloss overlaps between them. These systems perform better than the random baseline and one post--evaluation variation was within a respectable margin of the median result attained by all participating systems.
The present study uses a computational approach to examine the role of semantic constraints in normal reading. This methodology avoids confounds inherent in conventional measures of predictability, allowing for theoretically deeper accounts of semantic processing. We start from a definition of associations between words based on the significant log likelihood that two words co-occur frequently together in the sentences of a large text corpus. Direct associations between stimulus words were controlled, and semantic feature overlap between prime and target words was manipulated by their common a…
InsightFlow: LLM-Driven Synthesis of Patient Narratives for Mental Health into Causal Models
Department of Mathematics, IIT Delhi（印度德里理工学院数学系）；Department of Electrical Engineering, IIT Delhi（印度德里理工学院电气工程系）；Department of Computer Science and Engineering, IIIT Manipur（曼尼普尔理工学院计算机科学与工程系）
Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing and Engineering, Indiana University Bloomington（印第安纳大学伯明顿分校卢迪信息学、计算与工程学院复杂网络与系统研究中心）；Network Science Institute, Indiana University, Bloomington (IUNI)（印第安纳大学布卢明顿分校网络科学研究所）；Connection Science, Massachusetts Institute of Technology（麻省理工学院连接科学中心）；Sirius XM（天狼星XM公司）
The COVID-19 pandemic is a global crisis that has been testing every society and exposing the critical role of local politics in crisis response. In the United States, there has been a strong partisan divide between the Democratic and Republican party's narratives about the pandemic which resulted in polarization of individual behaviors and divergent policy adoption across regions. As shown in this case, as well as in most major social issues, strongly polarized narrative frameworks facilitate such narratives. To understand polarization and other social chasms, it is critical to dissect these …
Benchmarking LLMs on the Semantic Overlap Summarization Task
Research Center Trustworth Data Science and Security, Technical University of Dortmund（多特蒙德技术大学信任数据科学与安全研究中心）；School of Computing, Wichita State University（威斯康星州立大学计算机学院）；Department of Computer Science, NC State University（北卡罗来纳州立大学计算机科学系）
Name disambiguation has become one of the main themes in the Semantic Web agenda. The semantic web is an extension of the current Web in which information is not only given well-defined meaning, but also has many purposes that contain the ambiguous naturally or a lot of thing came with the overlap, mainly deals with the persons name. Therefore, we develop an approach to extract keywords from web snippet with utilizing the overlap principle, a concept to understand things with ambiguous, whereby features of person are generated for dealing with the variety of web, the web is steadily gaining gr…
