Ideology is at the core of political science research. Yet, there still does not exist general-purpose tools to characterize and predict ideology across different genres of text. To this end, we study Pretrained Language Models using novel ideology-driven pretraining objectives that rely on the comparison of articles on the same story written by media of different ideologies. We further collect a large-scale dataset, consisting of more than 3.6M political news articles, for pretraining. Our model POLITICS outperforms strong baselines and the previous state-of-the-art models on ideology prediction and stance detection tasks. Further analyses show that POLITICS is especially good at understanding long or formally written texts, and is also robust in few-shot learning scenarios.
Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Models
Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能和数据分析实验室）；King Abdullah University of Science and Technology (KAUST)（卡布斯科学技术大学）；University of Copenhagen（哥本哈根大学）；University of Macau（澳门大学）
Department of Linguistics, University of British Columbia（不列颠哥伦比亚大学语言学系）；Department of Political Science, University of Toronto Scarborough（多伦多大学斯卡伯勒分校政治学系）；Vector Institute, Department of Computer Science, University of Toronto（向量研究所、多伦多大学计算机科学系）
Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction
Department of Methodology and Statistics, Utrecht University（方法论与统计学系，乌特雷赫特大学）；Department of Languages, Literature and Communication, Utrecht University（语言、文学与传播系，乌特雷赫特大学）
Large language models (LLMs) are increasingly used in content moderation systems, where ensuring fairness and neutrality is essential. In this study, we examine how persona adoption influences the consistency and fairness of harmful content classification across different LLM architectures, model sizes, and content modalities (language vs. vision). At first glance, headline performance metrics suggest that personas have little impact on overall classification accuracy. However, a closer analysis reveals important behavioral shifts. Personas with different ideological leanings display distinct …
We investigate the impact of political ideology biases in training data. Through a set of comparison studies, we examine the propagation of biases in several widely-used NLP models and its effect on the overall retrieval accuracy. Our work highlights the susceptibility of large, complex models to propagating the biases from human-selected input, which may lead to a deterioration of retrieval accuracy, and the importance of controlling for these biases. Finally, as a way to mitigate the bias, we propose to learn a text representation that is invariant to political ideology while still judging t…
How Identity and Opinion Shape Political Sycophancy in LLMs
National Yang Ming Chiao Tung University（国立阳明交通大学）；Academia Sinica（中央研究院）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）
We analyze publicly available US Supreme Court documents using automated stance detection. In the first phase of our work, we investigate the extent to which the Court's public-facing language is political. We propose and calculate two distinct ideology metrics of SCOTUS justices using oral argument transcripts. We then compare these language-based metrics to existing social scientific measures of the ideology of the Supreme Court and the public. Through this cross-disciplinary analysis, we find that justices who are more responsive to public opinion tend to express their ideology during oral …
Against Political Polarization: A Unified Framework for Tracing Evolving Political Ideologies on Social Media
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Science and Technology of China（中国科学技术大学）；The Hong Kong University of Science and Technology（香港科技大学）
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Stony Brook University（石溪大学）
A news article's title, content and link structure often reveal its political ideology. However, most existing works on automatic political ideology detection only leverage textual cues. Drawing inspiration from recent advances in neural inference, we propose a novel attention based multi-view model to leverage cues from all of the above views to identify the ideology evinced by a news article. Our model draws on advances in representation learning in natural language processing and network science to capture cues from both textual content and the network structure of news articles. We empiric…
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）；Qatar Computing Research Institute, HBKU（卡塔尔计算研究所（哈马德·本·哈利法大学））
We explore the task of predicting the leading political ideology or bias of news articles. First, we collect and release a large dataset of 34,737 articles that were manually annotated for political ideology -left, center, or right-, which is well-balanced across both topics and media. We further use a challenging experimental setup where the test examples come from media that were not seen during training, which prevents the model from learning to detect the source of the target news article instead of predicting its political ideology. From a modeling perspective, we propose an adversarial m…
Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs
Institute for Natural Language Processing, University of Stuttgart（斯图加特大学自然语言处理研究所）；Faculty of Electrical Engineering and Computing, University of Zagreb（萨格勒布大学电气与计算机学院）；University of Manchester（曼彻斯特大学）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Pittsburgh（匹兹堡大学）；Rutgers University（罗格斯大学）
Analyzing ideology and polarization is of critical importance in advancing our grasp of modern politics. Recent research has made great strides towards understanding the ideological bias (i.e., stance) of news media along the left-right spectrum. In this work, we instead take a novel and more nuanced approach for the study of ideology based on its left or right positions on the issue being discussed. Aligned with the theoretical accounts in political science, we treat ideology as a multi-dimensional construct, and introduce the first diachronic dataset of news articles whose ideological positi…
Università degli Studi di Torino（都灵大学）；Universitat Politécnica de Valéncia（瓦伦西亚理工大学）
Stance detection, the task of identifying the speaker's opinion towards a particular target, has attracted the attention of researchers. This paper describes a novel approach for detecting stance in Twitter. We define a set of features in order to consider the context surrounding a target of interest with the final aim of training a model for predicting the stance towards the mentioned targets. In particular, we are interested in investigating political debates in social media. For this reason we evaluated our approach focusing on two targets of the SemEval-2016 Task6 on Detecting stance in tw…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Microsoft Research（微软研究院）
Participants in political discourse employ rhetorical strategies -- such as hedging, attributions, or denials -- to display varying degrees of belief commitments to claims proposed by themselves or others. Traditionally, political scientists have studied these epistemic phenomena through labor-intensive manual content analysis. We propose to help automate such work through epistemic stance prediction, drawn from research in computational semantics, to distinguish at the clausal level what is asserted, denied, or only ambivalently suggested by the author or other mentioned entities (belief hold…
TikStance: A Multimodal and Hierarchical Dataset for Multi-target Stance Analysis in TikTok Political Conversations
School of Artificial Intelligence, Shenzhen Technology University（深圳技术大学人工智能学院）；School of Cyber Science and Technology, University of Science and Technology of China（中国科学技术大学网络空间安全学院）
IBM Research AI（IBM研究院人工智能部门）；Queen’s University Belfast（贝尔法斯特女王大学）
Ideological leanings of an individual can often be gauged by the sentiment one expresses about different issues. We propose a simple framework that represents a political ideology as a distribution of sentiment polarities towards a set of topics. This representation can then be used to detect ideological leanings of documents (speeches, news articles, etc.) based on the sentiments expressed towards different topics. Experiments performed using a widely used dataset show the promise of our proposed approach that achieves comparable performance to other methods despite being much simpler and mor…
