arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.01423cs.CL
收藏

量化大型语言模型在多轮交互中的对话可靠性

Quantifying Conversational Reliability of Large Language Models under Multi-Turn Interaction

Jiyoon Myung

备注：
Accepted at the Workshop on Assessing and Improving Reliability of Foundation Models in the Real World (AAAI 2026)
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究评估了大型语言模型在多轮对话中的可靠性，发现其在复杂交互中存在显著下降，揭示了指令漂移、意图混淆等失败模式，强调了对LLM进行压力测试和改进评估方法的重要性。

摘要：

大型语言模型（LLMs）正被越来越多地应用于现实世界中，用户进行长时间、跨主题的对话，这些对话依赖于先前的上下文。然而，它们在现实中的多轮交互中的可靠性仍知之甚少。我们通过三个代表性任务系统地评估了对话可靠性，这些任务反映了实际交互挑战：（1）在话题转换中保持全局约束，（2）在交错意图中选择正确的工具或代理，（3）在修订和干扰下跟踪结构化实体。每个任务都配对单轮和多轮设置，使我们能够量化在延长对话中可靠性下降的程度。在商业和开源模型中，我们观察到显著的可靠性下降，尤其是较小的模型。错误分析揭示了反复出现的失败模式，如指令漂移、意图混淆和上下文覆盖，这些都会影响操作系统中的可靠行为。我们的发现强调了对LLM进行对话可靠性压力测试以及开发更稳健的评估方法以实现可信部署的必要性。

英文摘要：

Large Language Models (LLMs) are increasingly deployed in real-world applications where users engage in extended, mixed-topic conversations that depend on prior context. Yet, their reliability under realistic multi-turn interactions remains poorly understood. We conduct a systematic evaluation of conversational reliability through three representative tasks that reflect practical interaction challenges: (1) maintaining global constraints across topic shifts, (2) selecting the correct tool or agent amid interleaved intents, and (3) tracking structured entities under revisions and distractions. Each task pairs single-turn and multi-turn settings, allowing us to quantify reliability degradation under extended dialogue. Across both commercial and open-source models, we observe substantial declines in reliability, particularly for smaller models. Error analyses reveal recurring failure modes such as instruction drift, intent confusion, and contextual overwriting, which compromise dependable behavior in operational systems. Our findings highlight the need for stress-testing LLMs for conversational reliability and developing more robust evaluation methods for trustworthy deployment.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑