AI 中文总结
针对基于LLM系统回归测试的独特挑战，提出CASPER框架，通过进化切片识别与基于执行日志行为信息的优先级排序，实现更优的回归切片处理效果。
AI 中文摘要
基于大语言模型（LLM）的系统回归测试面临独特挑战，因为单个回归实例仅能提供有限的系统级回归信息。单个实例的失败不一定意味着存在有意义的回归，也无法提供足够信息来诊断受影响的行为。反之，仅基于整体系统性能变化检测回归的粒度太粗，无法确定哪些行为受到影响。这促使通过测试套件切片在中间层分析回归实例。为应对这一挑战，我们提出CASPER，一种用于基于LLM的系统中提示级和模型级变更的高效回归测试的感知变更的切片优先级排序框架。CASPER首先使用进化切片识别方法，识别包含语义相关且具有一致性能特征的实例的回归切片。在基于LLM的应用发生变更时，CASPER根据从执行日志中提取的行为信息，按切片的回归可能性对其进行优先级排序。我们在软件问题解决领域实例化CASPER，将切片识别与基于聚类的基线进行对比，将回归切片优先级排序与随机排名基线进行对比。结果显示，CASPER生成的切片一致性更高，同时保持相当或更优的语义连贯性，并在不同基于LLM的系统变更和测试预算下提升了回归切片优先级排序的效果。
英文摘要
Regression testing for LLM-based systems poses unique challenges because individual regression instances provide limited information about system-level regressions. A failure in a single instance does not necessarily indicate a meaningful regression or provide sufficient information to diagnose affected behaviors. Conversely, detecting regressions based only on overall system performance changes is too coarse-grained, as it does not identify which behaviors are affected. This motivates analyzing regression instances at an intermediate level through test suite slices. To address this challenge, we propose CASPER, a change-aware slice prioritization framework for efficient regression testing of prompt-level and model-level changes in LLM-based systems. CASPER first identifies regression slices containing semantically related instances with consistent performance characteristics using an evolutionary slice identification approach. Given a change to an LLM-based application, CASPER prioritizes slices according to their likelihood of regression using behavioral information extracted from execution logs. We instantiate CASPER in the software issue resolution domain and evaluate slice identification against clustering-based baselines and regressed slice prioritization against a random ranking baseline. Results show that CASPER generates more consistent slices while maintaining comparable or better semantic coherence and improves regressed slice prioritization across different LLM-based system changes and testing budgets.
