评估多轮LLM系统中模型切换导致的性能漂移
Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systems
AI总结:
研究评估了多轮LLM系统中模型切换导致的性能漂移,通过switch-matrix基准测试发现切换对对话质量有显著影响,并提出分解漂移因素以监控切换风险。
摘要:
在部署的多轮LLM系统中,由于升级、跨提供者路由和回退,模型经常在交互过程中切换。这种切换会引发上下文不匹配:后一轮的模型必须基于由不同模型生成的对话前缀进行条件生成,可能引发隐性的性能漂移。我们引入了一个switch-matrix基准测试,通过为早期轮次运行前缀模型和为最终轮次运行后缀模型,并使用配对的回合级bootstrap置信区间与无切换基线进行比较,来衡量这种影响。在CoQA对话问答和Multi-IF基准测试中,即使单轮切换也导致普遍且统计显著的方向性影响,并可能在Multi-IF严格成功率中使结果波动-8至+13个百分点,在CoQA中使F1分数波动±4分,与常见模型层级之间的无切换差距(例如GPT-5-nano vs GPT-5-mini)相当。我们进一步发现系统兼容性模式:某些后缀模型在几乎任何非自身对话历史下都会退化,而其他模型在几乎任何外来前缀下会提升。为了实现压缩的手动切换风险监控,我们将切换引起的漂移分解为每模型前缀影响和后缀易感性术语,这在基准测试中解释了约70%的方差。这些结果将切换鲁棒性定位为一个操作可靠性维度,单模型基准测试所遗漏的,推动了在多轮系统中进行显式监控和切换意识缓解的必要性。
英文摘要:
Deployed multi-turn LLM systems routinely switch models mid-interaction due to upgrades, cross-provider routing, and fallbacks. Such handoffs create a context mismatch: the model generating later turns must condition on a dialogue prefix authored by a different model, potentially inducing silent performance drift. We introduce a switch-matrix benchmark that measures this effect by running a prefix model for early turns and a suffix model for the final turn, and comparing against the no-switch baseline using paired episode-level bootstrap confidence intervals. Across CoQA conversational QA and Multi-IF benchmarks, even a single-turn handoff yields prevalent and statistically significant, directional effects and may swing outcomes by -8 to +13 percentage points in Multi-IF strict success rate and +/- 4 absolute F1 on CoQA, comparable to the no-switch gap between common model tiers (e.g., GPT-5-nano vs GPT-5-mini). We further find systematic compatibility patterns: some suffix models degrade under nearly any non-self dialogue history, while others improve under nearly any foreign prefix. To enable compressed handoff risk monitoring, we decompose switch-induced drift into per-model prefix influence and suffix susceptibility terms, accounting for ~70% of variance across benchmarks. These results position handoff robustness as an operational reliability dimension that single-model benchmarks miss, motivating explicit monitoring and handoff-aware mitigation in multi-turn systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。