arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.06182cs.CLcs.LG
收藏

了解Transformers用于Twitter中的Bot检测

Understanding Transformers for Bot Detection in Twitter

Expert.ai Research Lab(Expert.ai研究实验室)

Andres Garcia-Silva, Cristian Berrio, Jose Manuel Gomez-Perez

摘要:

在本文中,我们阐明了微调对社交媒体数据在神经语言模型内部表示中的影响。我们专注于Twitter中的机器人检测,这是减轻和抵消社交媒体中虚假信息和偏见自动传播的一项关键任务。我们研究了使用预先训练的语言模型来解决检测由机器人或人类帐户完全基于其内容生成的推文。与GLUE等基准测试的一般趋势不同,BERT在常规文本的大多数分类任务中通常优于GPT和GPT-2等生成Transformers,我们观察到,在机器人检测任务中微调生成Transformers会产生更高的准确性。我们分析了每个Transformer的架构组件,并研究了微调对它们的隐藏状态和输出表示的影响。在我们的研究结果中,我们表明,BERT在预训练过程中捕获的部分语法信息和分布特性在微调时丢失,而生成式预训练方法设法保留这些特性。

英文摘要:

In this paper we shed light on the impact of fine-tuning over social media data in the internal representations of neural language models. We focus on bot detection in Twitter, a key task to mitigate and counteract the automatic spreading of disinformation and bias in social media. We investigate the use of pre-trained language models to tackle the detection of tweets generated by a bot or a human account based exclusively on its content. Unlike the general trend in benchmarks like GLUE, where BERT generally outperforms generative transformers like GPT and GPT-2 for most classification tasks on regular text, we observe that fine-tuning generative transformers on a bot detection task produces higher accuracies. We analyze the architectural components of each transformer and study the effect of fine-tuning on their hidden states and output representations. Among our findings, we show that part of the syntactical information and distributional properties captured by BERT during pre-training is lost upon fine-tuning while the generative pre-training approach manage to preserve these properties.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑