像AI一样看待:LLMs如何应用(并误用)维基百科中立规范
Seeing Like an AI: How LLMs Apply (and Misapply) Wikipedia Neutrality Norms
AI总结:
研究评估LLMs在检测和纠正偏见维基百科编辑时的表现,发现其在中立性检测上准确率低,但生成任务表现较好,但存在额外非中立性修改,引发对社区规范执行与公众认知之间差异的思考。
摘要:
大型语言模型(LLMs)在广泛语料上进行训练后,被用于具有专门规范的社区。提供LLMs社区规则是否足以使模型遵循这些规范?我们评估LLMs根据维基百科中立观点(NPOV)政策检测(任务1)和纠正(任务2)偏见维基百科编辑的能力。LLMs在偏见检测上表现不佳,仅在平衡数据集上达到64%的准确率。模型表现出对比偏见(一些低估偏见,其他高估偏见),表明不同的中立先验假设。LLMs在生成任务中表现更好,删除了79%被维基百科编辑删除的单词。然而,LLMs在维基百科编辑更简单的中性化之外,做出了额外的修改,导致高召回但低精度的编辑。有趣的是,众包工人认为AI重写更中性(70%)且流畅(61%) than维基百科编辑重写。定性分析发现LLMs有时比维基百科编辑更全面地应用NPOV,但经常做出无关的非NPOV修改(如语法)。LLMs可能以与公众共鸣的方式应用规则,但与社区专家产生分歧。尽管在生成任务中可能有效,LLMs可能减少编辑者的自主权并增加 moderation 工作量(例如,验证添加)。即使规则易于表达,让LLMs像社区成员一样应用它们可能仍然困难。
英文摘要:
Large language models (LLMs) are trained on broad corpora and then used in communities with specialized norms. Is providing LLMs with community rules enough for models to follow these norms? We evaluate LLMs' capacity to detect (Task 1) and correct (Task 2) biased Wikipedia edits according to Wikipedia's Neutral Point of View (NPOV) policy. LLMs struggled with bias detection, achieving only 64% accuracy on a balanced dataset. Models exhibited contrasting biases (some under- and others over-predicted bias), suggesting distinct priors about neutrality. LLMs performed better at generation, removing 79% of words removed by Wikipedia editors. However, LLMs made additional changes beyond Wikipedia editors' simpler neutralizations, resulting in high-recall but low-precision editing. Interestingly, crowdworkers rated AI rewrites as more neutral (70%) and fluent (61%) than Wikipedia-editor rewrites. Qualitative analysis found LLMs sometimes applied NPOV more comprehensively than Wikipedia editors but often made extraneous non-NPOV-related changes (such as grammar). LLMs may apply rules in ways that resonate with the public but diverge from community experts. While potentially effective for generation, LLMs may reduce editor agency and increase moderation workload (e.g., verifying additions). Even when rules are easy to articulate, having LLMs apply them like community members may still be difficult.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。