Robots in the Middle: Evaluating LLMs in Dispute Resolution
University of Montreal(蒙特利尔大学);Maastricht University(马斯特里赫特大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Carnegie Mellon University(卡内基梅隆大学);Université Libre de Bruxelles(布鲁塞尔自由大学)
Jinzhe Tan, Hannes Westermann, Nikhil Reddy Pottanigari, Jaromír Šavelka, Sébastien Meeùs, Mia Godet, Karim Benyekhlef
Mediation is a dispute resolution method featuring a neutral third-party (mediator) who intervenes to help the individuals resolve their dispute. In this paper, we investigate to which extent large language models (LLMs) are able to act as mediators. We investigate whether LLMs are able to analyze dispute conversations, select suitable intervention types, and generate appropriate intervention messages. Using a novel, manually created dataset of 50 dispute scenarios, we conduct a blind evaluation comparing LLMs with human annotators across several key metrics. Overall, the LLMs showed strong performance, even outperforming our human annotators across dimensions. Specifically, in 62% of the cases, the LLMs chose intervention types that were rated as better than or equivalent to those chosen by humans. Moreover, in 84% of the cases, the intervention messages generated by the LLMs were rated as better than or equal to the intervention messages written by humans. LLMs likewise performed favourably on metrics such as impartiality, understanding and contextualization. Our results demonstrate the potential of integrating AI in online dispute resolution (ODR) platforms.
Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);Shihezi University(石河子大学);Beihang University(北航);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Jilin University(吉林大学);Central South University(中南大学);Jinan University(暨南大学)
E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Science and Technology of China(中国科学技术大学);Southern University of Science and Technology(南方科技大学);UNION INFORMATION(联信信息);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
The Biased Samaritan: LLM biases in Perceived Kindness
Department of Cognitive Science University of California, Davis(认知科学系加州大学戴维斯分校);Department of Statistics University of California, Davis(统计学系加州大学戴维斯分校);Department of Computer Science University of California, Davis(计算机科学系加州大学戴维斯分校)