MARS-Bench：用于对话评估的多轮真实体育场景基准
MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation
AI总结：
提出MARS-Bench基准，通过赛况解说构建真实多轮对话，评估LLMs在超长多轮、交互多轮及跨轮任务中的鲁棒性，并揭示闭源模型优势、显式推理提升及注意力汇聚致性能下降机制。
摘要：
大型语言模型（LLMs），例如ChatGPT，已被广泛应用于真实对话场景中。然而，LLMs的鲁棒性，尤其是在处理包含频繁动机转移和复杂跨轮依赖的长复杂对话会话方面，一直备受批评。尽管如此，现有的基准无法完全反映这些弱点。我们提出了MARS-Bench，一个多轮真实体育场景对话基准，旨在弥补这一空白。MARS-Bench从详细赛况文本解说构建，具有真实对话特征，专门设计用于评估多轮对话的三个关键方面：超长多轮、交互式多轮和跨轮任务。在MARS-Bench上的大量实验还表明，闭源LLMs显著优于开源替代方案，显式推理显著提升了LLMs处理长复杂对话会话的鲁棒性，并且LLMs在处理动机转移和复杂跨轮依赖时确实面临重大挑战。此外，我们基于Qwen2.5-7B-Instruction中的注意力可视化实验，提供了关于特殊Token导致的注意力汇聚如何引发LLMs在处理长复杂对话会话时性能下降的机制可解释性。
英文摘要：
Large Language Models (\textbf{LLMs}), e.g. ChatGPT, have been widely adopted in real-world dialogue applications. However, LLMs' robustness, especially in handling long complex dialogue sessions, including frequent motivation transfer, sophisticated cross-turn dependency, is criticized all along. Nevertheless, no existing benchmarks can fully reflect these weaknesses. We present \textbf{MARS-Bench}, a \textbf{M}ulti-turn \textbf{A}thletic \textbf{R}eal-world \textbf{S}cenario Dialogue \textbf{Bench}mark, designed to remedy the gap. MARS-Bench is constructed from play-by-play text commentary so to feature realistic dialogues specifically designed to evaluate three critical aspects of multi-turn conversations: Ultra Multi-turn, Interactive Multi-turn, and Cross-turn Tasks. Extensive experiments on MARS-Bench also reveal that closed-source LLMs significantly outperform open-source alternatives, explicit reasoning significantly boosts LLMs' robustness on handling long complex dialogue sessions, and LLMs indeed face significant challenges when handling motivation transfer and sophisticated cross-turn dependency. Moreover, we provide mechanistic interpretability on how attention sinks due to special tokens lead to LLMs' performance degradation when handling long complex dialogue sessions based on attention visualization experiment in Qwen2.5-7B-Instruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
