从文章到发布者：聚合语言模型预测用于新闻来源可靠性推断
From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference
浏览论文内容
中文总结 AI 辅助
本研究提出两阶段框架，通过聚合文章级语言模型预测推断新闻发布者可靠性，在严格不相交协议下将准确率从0.60提升至0.69，并发现政治倾向与误分类相关。
中文摘要 AI 辅助
传统上，新闻发布者的可靠性由专家组织评估，这些组织在来源层面评价编辑实践、透明度和事实标准。当这一过程转化为计算方法时，问题通常被设定在单篇文章层面，模型在一组预标注的文章上训练，并在测试阶段评估其性能。在本工作中，我们将新闻来源可靠性推断作为来源层面的预测问题进行研究。我们提出一个两阶段框架，其中基于Transformer的语言模型首先估计单篇文章的可靠性，随后聚合文章层面的预测以推断未见过的发布者的可靠性。为逼近真实部署条件，我们强制执行严格的发布者不相交评估协议，确保没有任何发布者同时出现在训练集和测试集中。在来自439个英语发布者的19,476篇政治新闻文章上进行的实验（这些文章带有NewsGuard可靠性评级标签）表明，聚合显著提升了鲁棒性和性能，将准确率从文章层面的大约0.60提升至发布者层面的0.69。最后，我们分析了预测错误如何随政治倾向变化，揭示了政治立场与误分类模式之间存在统计上显著的关联。总体而言，我们的发现表明，仅通过聚合文本信号即可推断发布者可靠性，支持可扩展且基于内容的自动化新闻来源评估方法。
英文摘要
Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual standards at source. When this process is translated into a computational approach, the problem is often formulated at the level of individual articles, with models being trained on a set of pre-labelled articles and their performance being evaluated in a test phase. In this work, we investigate news source reliability inference as a source-level prediction problem. We propose a two-stage framework in which transformer-based language models first estimate the reliability of individual articles and subsequently aggregate article-level predictions to infer the reliability of previously unseen publishers. To approximate realistic deployment conditions, we enforce a strict publisher-disjoint evaluation protocol, ensuring that no publisher appears in both training and test sets. Experiments on 19,476 political news articles from 439 English-language publishers labeled with NewsGuard reliability ratings show that aggregation substantially improves robustness and performance, increasing accuracy from approximately 0.60 at the article level to 0.69 at the publisher level. Finally, we analyze how prediction errors vary across political orientations, revealing statistically significant associations between political leaning and misclassification patterns. Overall, our findings show that publisher reliability can be inferred from aggregated textual signals alone, supporting scalable and content-based approaches to automated news source assessment.
发表机构
- IMT School for Advanced Studies Lucca（卢卡IMT高等研究院）
- Institute of Informatics and Telematics, National Research Council (IIT-CNR)（国家研究委员会信息学与远程信息处理研究所（IIT-CNR））
机构由 AI 辅助整理，请以论文原文为准。
