arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.03508cs.CL
收藏

一战接一战:通过一个演进框架探测LLMs在多轮指令遵循中的极限

One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Jilin University(吉林大学)

Qi Jia, Ye Shen, Xiujie Song, Kaiwei Zhang, Shibo Wang, Dun Pei, Xiangyang Zhu, Guangtao Zhai

AI总结:

本文提出EvolIF演进框架,通过多轮对话测试LLMs的指令遵循能力,揭示失败恢复和细粒度指令遵循的不足,GPT-5在鲁棒性上表现最佳。

摘要:

评估LLMs在多主题对话中的指令遵循能力是至关重要的,但具有挑战性。现有的基准测试仅限于固定数量的回合,容易饱和且未能考虑用户的交互体验。在本文中,我们提出了一种新的框架,该框架具有三层跟踪机制和一个查询合成代理,以模拟连续的用户行为。基于流理论,我们引入了以过程为中心的度量标准,并在耗尽用户耐心时才终止对话评估。利用这一框架,我们提出了EvolIF,一个覆盖12个约束组的演进基准测试。我们的分析揭示了失败恢复和细粒度指令遵循的不足,随着对话深度的增加,性能分层变得明显。GPT-5展示了最持久的韧性,保持66.40%的鲁棒性得分,优于Gemini-3-Pro 5.59个百分点,而其他模型则落后。数据和代码将在https://github.com/JiaQiSJTU/EvolIF上发布。

英文摘要:

Evaluating LLMs' instruction-following ability in multi-topic dialogues is essential yet challenging. Existing benchmarks are limited to a fixed number of turns, susceptible to saturation and failing to account for users' interactive experience. In this work, we propose a novel framework featuring a three-layer tracking mechanism and a query synthesis agent to mimic sequential user behaviors. Grounded in Flow Theory, we introduce process-centric metrics and terminate a conversational evaluation only upon exhausting user patience. Leveraging this framework, we present EvolIF, an evolving benchmark covering 12 constraint groups. Our analysis reveals deficiencies in failure recovery and fine-grained instruction following, with performance stratification becoming evident as conversational depth increases. GPT-5 demonstrates the most sustained resilience, maintaining a 66.40% robustness score, outperforming Gemini-3-Pro by 5.59%, while other models lag behind. Data and code will be released at https://github.com/JiaQiSJTU/EvolIF.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑