arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.17399cs.CLcs.AI
收藏

MultiChallenge:一个挑战前沿 LLMs 的现实多轮对话评估基准

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

Scale AI

Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, Chen Xing

AI总结:

提出 MultiChallenge 基准,通过四类需同时考察指令遵循、上下文分配与推理的挑战评估 LLMs 的多轮对话能力,并采用 LLM as judge 自动评估,揭示前沿模型在该基准上准确率均低于 50%。

摘要:

我们提出了 MultiChallenge,这是一个评估大型语言模型(LLMs)与人类用户进行多轮对话能力的开创性基准,该能力对其应用至关重要却尚未得到充分检验。MultiChallenge 识别了多轮对话中的四类挑战,这些挑战在当前人类与 LLM 交互中既常见又现实,同时也对所有当前前沿 LLMs 构成挑战。所有 4 项挑战均要求同时具备准确的指令遵循、上下文分配和上下文推理能力。我们还开发了带有实例级评分标准的 LLM as judge,以促成一种与经验丰富的人类评估者达成公平一致的自动评估方法。尽管在现有的多轮评估基准上取得了近乎完美的分数,所有前沿模型在 MultiChallenge 上的准确率均低于 50%,其中表现最佳的 Claude 3.5 Sonnet(2024年6月)平均准确率仅为 41.4%。

英文摘要:

We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑