提出 MultiChallenge 基准,通过四类需同时考察指令遵循、上下文分配与推理的挑战评估 LLMs 的多轮对话能力,并采用 LLM as judge 自动评估,揭示前沿模型在该基准上准确率均低于 50%。
摘要:
我们提出了 MultiChallenge,这是一个评估大型语言模型(LLMs)与人类用户进行多轮对话能力的开创性基准,该能力对其应用至关重要却尚未得到充分检验。MultiChallenge 识别了多轮对话中的四类挑战,这些挑战在当前人类与 LLM 交互中既常见又现实,同时也对所有当前前沿 LLMs 构成挑战。所有 4 项挑战均要求同时具备准确的指令遵循、上下文分配和上下文推理能力。我们还开发了带有实例级评分标准的 LLM as judge,以促成一种与经验丰富的人类评估者达成公平一致的自动评估方法。尽管在现有的多轮评估基准上取得了近乎完美的分数,所有前沿模型在 MultiChallenge 上的准确率均低于 50%,其中表现最佳的 Claude 3.5 Sonnet(2024年6月)平均准确率仅为 41.4%。
英文摘要:
We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.
MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Department of Computer Science and Software Engineering, The University of Western Australia(西澳大学计算机科学与软件工程系)
A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Health
School of Computer Science, Faculty of Engineering(工程学院计算机科学学院);The University of Sydney(悉尼大学);School of Computer Science and Engineering(计算机科学与工程学院);University of New South Wales(新南威尔士大学);School of Optometry and Vision Science(视光与视觉科学学院)
BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Virginia Tech(弗吉尼亚理工大学);Bangladesh University of Engineering and Technology(孟加拉工程与技术大学);BRAC University(BRAC大学);University of Toronto(多伦多大学)
SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
Rensselaer Polytechnic Institute(拉特格斯理工学院);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Pacific Northwest National Laboratory(太平洋西北国家实验室);National Renewable Energy Laboratory(国家可再生能源实验室)
TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning
University of Oxford(牛津大学);VulpiVox Intelligence;Eindhoven University of Technology(埃因霍温理工大学);Griffith University(格里菲斯大学);Squirrel Ai Learning;East China Normal University(华东师范大学)
AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment
Department of Computer Science, Columbia University, New York, NY, USA(计算机科学系,哥伦比亚大学,纽约,纽约州,美国);Department of Biomedical Informatics, Columbia University, New York, NY, USA(生物医学信息学系,哥伦比亚大学,纽约,纽约州,美国);Department of Emergency Medicine, Columbia University Irving Medical Center, New York, NY, USA(急诊医学系,哥伦比亚大学伊文思医疗中心,纽约,纽约州,美国)
Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs
Unbabel;Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico & Universidade de Lisboa (Lisbon ELLIS Unit)(里斯本大学技术学院及里斯本大学(里斯本ELLIS单位));MICS, CentraleSupélec, Université Paris-Saclay(MICS,CentraleSupélec,巴黎萨克雷大学)
It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics
Université de Montréal, MILA(蒙特利尔大学,MILA);Carnegie Mellon University(卡内基梅隆大学);MIT, Center for Research and Teaching in Economics(麻省理工学院,经济研究与教学中心);Cornell University, University of Regina(康奈尔大学, Regina大学);Cornell University, MIT(康奈尔大学,麻省理工学院)
Single-turn emergency psychiatric triage across 15 frontier AI chatbots
Max Planck UCL Centre for Computational Psychiatry and Ageing Research(马克斯·普朗克大学伦敦计算精神病学与衰老研究中心);UK AI Security Institute(英国人工智能安全研究所);Department of Experimental Psychology, University of Oxford(牛津大学实验心理学系);Microsoft AI(微软人工智能);Department of Psychiatry, University of Oxford(牛津大学精神病学系)