We present the task of modeling information propagation in literature, in which we seek to identify pieces of information passing from character A to character B to character C, only given a description of their activity in text. We describe a new pipeline for measuring information propagation in this domain and publish a new dataset for speaker attribution, enabling the evaluation of an important component of this pipeline on a wider range of literary texts than previously studied. Using this pipeline, we analyze the dynamics of information propagation in over 5,000 works of fiction, finding that information flows through characters that fill structural holes connecting different communities, and that characters who are women are depicted as filling this role much more frequently than characters who are men.
It is an open question to what extent perceptions of literary quality are derived from text-intrinsic versus social factors. While supervised models can predict literary quality ratings from textual factors quite successfully, as shown in the Riddle of Literary Quality project (Koolen et al., 2020), this does not prove that social factors are not important, nor can we assume that readers make judgments on literary quality in the same way and based on the same information as machine learning models. We report the results of a pilot study to gauge the effect of textual features on literary ratin…
Understanding Social Structures from Contemporary Literary Fiction using Character Interaction Graph -- Half Century Chronology of Influential Bengali Writers
The Pennsylvania State University（宾夕法尼亚州立大学）；Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）
Southern University of Science and Technology（南方科技大学）
Digital humanities is an important subject because it enables developments in history, literature, and films. In this paper, we perform an empirical study of a Chinese historical text, Records of the Three Kingdoms (\textit{Records}), and a historical novel of the same story, Romance of the Three Kingdoms (\textit{Romance}). We employ natural language processing techniques to extract characters and their relationships. Then, we characterize the social networks and sentiments of the main characters in the historical text and the historical novel. We find that the social network in \textit{Roman…
Universität Zürich（苏黎世大学）；Julius-Maximilians-Universität Würzburg（维尔茨堡朱利叶斯-马克西米利安大学）
Natural Language Processing and Machine Learning have considerably advanced Computational Literary Studies. Similarly, the construction of co-occurrence networks of literary characters, and their analysis using methods from social network analysis and network science, have provided insights into the micro- and macro-level structure of literary texts. Combining these perspectives, in this work we study character networks extracted from a text corpus of J.R.R. Tolkien's Legendarium. We show that this perspective helps us to analyse and visualise the narrative style that characterises Tolkien's w…
Computational thematics: Comparing algorithms for clustering the genres of literary fiction
Max Planck Institute for Evolutionary Anthropology（马克斯·普朗克进化人类学研究所）；Max Planck Institute for Geoanthropology（马克斯·普朗克地理人类学研究所）；Institute of Polish Language, Polish Academy of Sciences（波兰科学院波兰语言研究所）
This paper investigates the research question if senders of large amounts of irrelevant or unsolicited information - commonly called "spammers" - distort the network structure of social networks. Two large social networks are analyzed, the first extracted from the Twitter discourse about a big telecommunication company, and the second obtained from three years of email communication of 200 managers working for a large multinational company. This work compares network robustness and the stability of centrality and interaction metrics, as well as the use of language, after removing spammers and …
University of Michigan（密歇根大学）；University of New Mexico（新墨西哥大学）；Emory University（埃默里大学）；University of Washington（华盛顿大学）；Allen Institute for AI（艾伦人工智能研究所）
Language models increasingly rely on massive web dumps for diverse text data. However, these sources are rife with undesirable content. As such, resources like Wikipedia, books, and newswire often serve as anchors for automatically selecting web text most suitable for language modeling, a process typically referred to as quality filtering. Using a new dataset of U.S. high school newspaper articles -- written by students from across the country -- we investigate whose language is preferred by the quality filter used for GPT-3. We find that newspapers from larger schools, located in wealthier, e…
Riveter: Measuring Power and Social Dynamics Between Entities
Allen Institute for AI（艾伦人工智能研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）；Emory University（埃默里大学）
Fairness is a principal social value that can be observed in civilisations around the world. A manifestation of this is in social agreements, often described in texts, such as contracts. Yet, despite the prevalence of such, a fairness metric for texts describing a social act remains wanting. To address this, we take a step back to consider the problem based on first principals. Instead of using rules or templates, we utilise social psychology literature to determine the principal factors that humans use when making a fairness assessment. We then attempt to digitise these using word embeddings …
Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs
University of Turku（图尔库大学）；ELLIS Institute Finland（芬兰ELLIS研究所）
Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks. However, there is ample evidence that they use the cultural biases that are undoubtedly present in the corpora they are trained on, implicitly creating harm with biased representations. To measure some forms of social bias in language models against protected demographic groups in the US, we introduce the Crowdsourced Stereotype Pairs benchmark (CrowS-Pairs). CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pai…
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
University of Vermont（佛蒙特大学）；Charles River Analytics（查尔斯河分析公司）
Working from a dataset of 118 billion messages running from the start of 2009 to the end of 2019, we identify and explore the relative daily use of over 150 languages on Twitter. We find that eight languages comprise 80% of all tweets, with English, Japanese, Spanish, and Portuguese being the most dominant. To quantify social spreading in each language over time, we compute the 'contagion ratio': The balance of retweets to organic messages. We find that for the most common languages on Twitter there is a growing tendency, though not universal, to retweet rather than share new content. By the e…
Readers' responses to literature have received scant attention in computational literary studies. The rise of social media offers an opportunity to capture a segment of these responses while data-driven analysis of these responses can provide new critical insight into how people "read". Posts discussing an individual book on Goodreads, a social media platform that hosts user discussions of popular literature, are referred to as "reviews", and consist of plot summaries, opinions, quotes, or some mixture of these. Since these reviews are written by readers, computationally modeling them allows o…
Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）智能科学与工程学院）；School of Information Science and Engineering, Ningbo University（宁波大学信息科学与工程学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
Boston University（波士顿大学）；Harvard University（哈佛大学）；Koç University（科奇大学）
Large language models trained on a mixture of NLP tasks that are converted into a text-to-text format using prompts, can generalize into novel forms of language and handle novel tasks. A large body of work within prompt engineering attempts to understand the effects of input forms and prompts in achieving superior performance. We consider an alternative measure and inquire whether the way in which an input is encoded affects social biases promoted in outputs. In this paper, we study T0, a large-scale multi-task text-to-text language model trained using prompt-based learning. We consider two di…
Halmstad University（哈尔姆斯塔德大学）；University of Tartu（塔尔图大学）
Establishing authorship of online texts is fundamental to combat cybercrimes. Unfortunately, text length is limited on some platforms, making the challenge harder. We aim at identifying the authorship of Twitter messages limited to 140 characters. We evaluate popular stylometric features, widely used in literary analysis, and specific Twitter features like URLs, hashtags, replies or quotes. We use two databases with 93 and 3957 authors, respectively. We test varying sized author sets and varying amounts of training/test texts per author. Performance is further improved by feature combination v…
Machines in the Crowd? Measuring the Footprint of Machine-Generated Text on Reddit
DIMES Dept., University of Calabria Italy；IT University of Copenhagen\ Centre for AI Denmark；DIMES Dept., University of Calabria；IT University of Copenhagen\ Centre for AI
Researchers have devised numerous ways to quantify social biases vested in pretrained language models. As some language models are capable of generating coherent completions given a set of textual prompts, several prompting datasets have been proposed to measure biases between social groups -- posing language generation as a way of identifying biases. In this opinion paper, we analyze how specific choices of prompt sets, metrics, automatic tools and sampling strategies affect bias results. We find out that the practice of measuring biases through text completion is prone to yielding contradict…
Amazon Alexa AI-NU（亚马逊Alexa人工智能-西北大学合作团队）；UC Santa Barbara（加州大学圣巴巴拉分校）；UCLA（加州大学洛杉矶分校）
Recent advances in deep learning techniques have enabled machines to generate cohesive open-ended text when prompted with a sequence of words as context. While these models now empower many downstream applications from conversation bots to automatic storytelling, they have been shown to generate texts that exhibit social biases. To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23,679 English text generation prompts for bias benchmarking across fiv…
Measuring Dimensions of Self-Presentation in Twitter Bios and their Links to Misinformation Sharing
University at Buffalo（布法罗大学）；GESIS Leibniz Institute for the Social Sciences（GESIS莱布尼茨社会科学研究所）；University of Pittsburgh（匹兹堡大学）；George Washington University（乔治华盛顿大学）；Northeastern University（东北大学）
VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models
University of Southern California（南加州大学）；University of Bristol（布里斯托大学）；Centre National de la Recherche Scientifique（法国国家科学研究中心）；University of California, Los Angeles（加州大学洛杉矶分校）
