arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.11936cs.AIcs.HC
收藏

使用大型语言模型估计在线审议中的贡献质量

Estimating Contribution Quality in Online Deliberations Using a Large Language Model

Center on Democracy, Development and the Rule of Law(民主、发展与法治中心);Stanford University(斯坦福大学);Management Science and Engineering(管理科学与工程学院);Cadence Design Systems(楷登电子公司)

Lodewijk Gelauff, Mohak Goyal, Bhargav Dindukurthi, Ashish Goel, Alice Siu

AI总结:

本文提出使用LLM替代人工评估在线审议中的贡献质量,模型在1-5分四项指标上优于个体标注员,接近三人组水平,并验证了助推能在不降质的前提下增加发言量。

摘要:

审议涉及参与者交换知识、论点和观点,已被证明在解决两极分化方面是有效的。Stanford Online Deliberation Platform促进了大规模审议。它支持小组就结构化议程进行基于视频的在线讨论,无需人类主持人。本文数据来自多次审议活动,包括一次与Meta合作在32个国家进行的活动,以及另一次与美国38所高等教育机构进行的活动。估计对话中贡献的质量对于评估特征和干预影响至关重要。传统上,这由人类标注员完成,既耗时又昂贵。我们使用大型语言模型(LLM)和八名人类标注员,基于理由、新颖性、对话的扩展以及进一步扩展的潜力对贡献进行评分,分数范围为1至5。标注员还为其评分提供简短理由。以其他人类标注员的平均评分作为真实标签,我们发现模型的表现优于个体人类标注员。虽然成对人类标注员在评分理由上优于模型,三人组在所有四个指标上均优于模型,但模型仍具竞争力。我们通过评估助推对审议质量的影响,展示了自动质量评分的实用性。我们首先观察到,在长时间不活动期间,个体助推非常有效,使个体在接下来30秒内请求发言的可能性增加了65%。使用我们的自动质量估计,我们表明由助推触发的消息的质量评分与未使用助推时相似,这表明助推在不损失整体质量的情况下,为对话产生了更多想法。

英文摘要:

Deliberation involves participants exchanging knowledge, arguments, and perspectives and has been shown to be effective at addressing polarization. The Stanford Online Deliberation Platform facilitates large-scale deliberations. It enables video-based online discussions on a structured agenda for small groups without requiring human moderators. This paper's data comes from various deliberation events, including one conducted in collaboration with Meta in 32 countries, and another with 38 post-secondary institutions in the US. Estimating the quality of contributions in a conversation is crucial for assessing feature and intervention impacts. Traditionally, this is done by human annotators, which is time-consuming and costly. We use a large language model (LLM) alongside eight human annotators to rate contributions based on justification, novelty, expansion of the conversation, and potential for further expansion, with scores ranging from 1 to 5. Annotators also provide brief justifications for their ratings. Using the average rating from other human annotators as the ground truth, we find the model outperforms individual human annotators. While pairs of human annotators outperform the model in rating justification and groups of three outperform it on all four metrics, the model remains competitive. We illustrate the usefulness of the automated quality rating by assessing the effect of nudges on the quality of deliberation. We first observe that individual nudges after prolonged inactivity are highly effective, increasing the likelihood of the individual requesting to speak in the next 30 seconds by 65%. Using our automated quality estimation, we show that the quality ratings for statements prompted by nudging are similar to those made without nudging, signifying that nudging leads to more ideas being generated in the conversation without losing overall quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑