arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.08550cs.CL
收藏

细粒度的捷克新闻文章数据集:可信度分析的跨学科方法

Fine-grained Czech News Article Dataset: An Interdisciplinary Approach to Trustworthiness Analysis

Charles University(查理大学);University College London(伦敦大学学院)

Matyáš Boháček, Michal Bravanský, Filip Trhlík, Václav Moravec

摘要:

我们提出了Verifee数据集:一个新的数据集的新闻文章与细粒度的可信度注释。我们开发了一种详细的方法,根据其参数评估文本,包括编辑透明度,记者惯例和客观报道,同时惩罚操纵技术。我们带来了来自社会,媒体和计算机科学的各种研究人员,以克服这一跨学科问题的障碍和有限的框架。我们从几乎$60$捷克在线新闻来源收集超过$10,000$独特的文章。在我们提出的可信度范围内,这些文章被归类为$4$类,从完全值得信赖的文章一直到操纵性文章。我们提供详细的统计数据,并研究整个集合中出现的趋势。最后,我们使用我们的数据集在可信度分类任务上微调了多个流行的序列到序列语言模型,并报告了$0.52$的最佳测试F-1得分。我们在https://verifee.ai/research上开源了数据集、注释方法和注释者的完整说明,以方便构建工作。我们相信类似的方法可以帮助防止虚假信息和教育领域的媒体素养。

英文摘要:

We present the Verifee Dataset: a novel dataset of news articles with fine-grained trustworthiness annotations. We develop a detailed methodology that assesses the texts based on their parameters encompassing editorial transparency, journalist conventions, and objective reporting while penalizing manipulative techniques. We bring aboard a diverse set of researchers from social, media, and computer sciences to overcome barriers and limited framing of this interdisciplinary problem. We collect over $10,000$ unique articles from almost $60$ Czech online news sources. These are categorized into one of the $4$ classes across the credibility spectrum we propose, raging from entirely trustworthy articles all the way to the manipulative ones. We produce detailed statistics and study trends emerging throughout the set. Lastly, we fine-tune multiple popular sequence-to-sequence language models using our dataset on the trustworthiness classification task and report the best testing F-1 score of $0.52$. We open-source the dataset, annotation methodology, and annotators' instructions in full length at https://verifee.ai/research to enable easy build-up work. We believe similar methods can help prevent disinformation and educate in the realm of media literacy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑