arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.07053cs.HCcs.CL
收藏

居中的机器人:评估大语言模型在纠纷解决中的表现

Robots in the Middle: Evaluating LLMs in Dispute Resolution

University of Montreal(蒙特利尔大学);Maastricht University(马斯特里赫特大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Carnegie Mellon University(卡内基梅隆大学);Université Libre de Bruxelles(布鲁塞尔自由大学)

Jinzhe Tan, Hannes Westermann, Nikhil Reddy Pottanigari, Jaromír Šavelka, Sébastien Meeùs, Mia Godet, Karim Benyekhlef

AI总结:

该研究探究大语言模型充当纠纷调解员的能力,通过含50个纠纷场景的数据集开展盲评,发现LLMs在干预类型选择、信息生成及公正性等多维度表现优于或等同人类,展现了其在在线纠纷解决平台的应用潜力。

摘要:

调解是一种纠纷解决方式,其特点是由中立的第三方(调解员)介入,帮助当事人解决纠纷。本文研究了大语言模型(LLMs)能够在多大程度上充当调解员。我们探究了LLMs是否能够分析纠纷对话、选择合适的干预类型并生成恰当的干预信息。我们利用一个全新的、人工构建的包含50个纠纷场景的数据集,开展了一项盲评实验,在多个关键指标上对比LLMs与人类标注者的表现。总体而言,LLMs表现出色,甚至在各维度上都优于我们的人类标注者。具体来说,在62%的案例中,LLMs选择的干预类型被评为优于或等同于人类选择的干预类型。此外,在84%的案例中,LLMs生成的干预信息被评为优于或等同于人类撰写的干预信息。LLMs在公正性、理解力和语境化等指标上同样表现优异。我们的研究结果证明了将人工智能集成到在线纠纷解决(ODR)平台中的潜力。

英文摘要:

Mediation is a dispute resolution method featuring a neutral third-party (mediator) who intervenes to help the individuals resolve their dispute. In this paper, we investigate to which extent large language models (LLMs) are able to act as mediators. We investigate whether LLMs are able to analyze dispute conversations, select suitable intervention types, and generate appropriate intervention messages. Using a novel, manually created dataset of 50 dispute scenarios, we conduct a blind evaluation comparing LLMs with human annotators across several key metrics. Overall, the LLMs showed strong performance, even outperforming our human annotators across dimensions. Specifically, in 62% of the cases, the LLMs chose intervention types that were rated as better than or equivalent to those chosen by humans. Moreover, in 84% of the cases, the intervention messages generated by the LLMs were rated as better than or equal to the intervention messages written by humans. LLMs likewise performed favourably on metrics such as impartiality, understanding and contextualization. Our results demonstrate the potential of integrating AI in online dispute resolution (ODR) platforms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑