发表机构
Athens University of Economics and Business; Archimedes, Athena Research Center（雅典经济与商业大学; 阿基米德雅典娜研究中心）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究创建PEFK语料库，对比人类与LLM的在线讨论促进倾向，发现LLM过度急于促进，训练ModernBert分类器修正效果优于LLM替代设置。
AI 中文摘要
在线讨论的自动化促进是一个长期存在的社会关切，因为我们在在线空间花费的时间越来越多，且内容审核方法存在不足。尽管已有关于如何促进讨论的研究，但尚未有人回答何时应进行促进这一关键问题。一个潜在答案是使用大语言模型（LLM），其表面上使自动化、大规模干预愈发可行。本研究通过定义促进的内涵、观察人类决定促进的时机，并将其决策与LLM的决策进行比较，探究LLM决定促进的时机。为此，我们创建了PEFK语料库，该语料库对所有相关促进数据集进行标准化和聚合。我们首次开展关于促进时机的调查，采用专业促进参与者和LLM作为评判模型执行调查。我们发现，人类更为谨慎，而LLM则过度急于进行促进；尽管两者在判断无需促进时都更确定。随后，我们研究是否可通过LLM的替代设置以及在现有数据集上训练ModernBert分类器来修正此行为，发现后者比前者表现更可靠，不过当前数据集带来的性能上限相对较低。
英文摘要
Automating facilitation in online discussions is a long-standing social concern given the increasing time we spend on online spaces and the failure of content moderation approaches. While studies have been conducted on how to facilitate, none have answered the essential question of when to do so. A potential answer is using LLMs, which ostensibly make automated, large-scale intervention increasingly feasible. In this study, we examine when LLMs decide to facilitate by defining what facilitation is, observing when humans decide to facilitate, and comparing their decisions with those made by LLMs. To this end, we create PEFK, a corpus standardizing and aggregating all relevant facilitation datasets. We are the first to run a survey on facilitation timing, which we execute using expert facilitative participants and LLM-as-a-judge models. We discover that while humans are more cautious, LLMs are excessively eager to facilitate, although both are more certain when judging that facilitation is not needed. We then investigate whether this behavior can be corrected using alternative setups for LLMs and training ModernBert classifiers on established datasets, finding that the latter perform more reliably than the former, although current datasets impose a relatively low performance ceiling.
CommentsFor the moderators: The acronym package may complain that some "acro" references are undefined. These references are, in fact, defined and the readability of the article remains the same
