重新思考人工智能/机器学习系统的问题解决
Rethinking Issue Resolution for AI/ML Systems
浏览论文内容
中文总结 AI 辅助
研究针对人工智能/机器学习系统问题解决，对四个系统的问题报告等做定性研究，发现其问题解决涉及多阶段反复活动、迭代实验等，还存在相关挑战，进而提出问题解决框架愿景及所需研究方向与工具支持。
中文摘要 AI 辅助
我们倡导针对维护工作流程及现代人工智能/机器学习系统特性量身定制的人工智能/机器学习问题解决框架。现有问题解决框架主要源于传统软件维护实践，未明确考虑人工智能/机器学习系统的常见特性，如随机行为、实验驱动工作流程及源代码之外的异构工件。为识别人工智能/机器学习系统中问题解决的独特特性并推动定制框架的需求，我们对四个广泛使用的人工智能/机器学习系统（TensorFlow、scikit - learn、MLflow和AutoGPT）中100个问题报告和拉取请求记录的问题解决工作流程进行了定性研究。我们的发现表明，人工智能/机器学习系统中的问题解决涉及跨越多个解决阶段的反复人工智能/机器学习相关活动、迭代实验和自适应验证，以及跨数据集、提示和模型配置等工件的协调更改。我们还观察到与可重复性、非确定性行为和工件协调相关的挑战。基于这些发现，我们提出了人工智能/机器学习问题解决框架的愿景，并讨论了实现这一愿景所需的研究方向和工具支持。
英文摘要
We advocate for AI/ML issue resolution frameworks tailored to maintenance workflows and the nature of modern AI/ML systems. Existing issue resolution frameworks largely emerged for traditional software maintenance practices and do not explicitly account for characteristics common in AI/ML systems, such as stochastic behavior, experimentation-driven workflows, and heterogeneous artifacts beyond source code. To identify the unique characteristics of issue resolution in AI/ML systems and motivate the need for tailored frameworks, we conducted a qualitative study of issue resolution workflows documented in 100 issue reports and pull requests across four widely used AI/ML systems: TensorFlow, scikit-learn, MLflow, and AutoGPT. Our findings suggest that issue resolution in AI/ML systems involves: recurring AI/ML-related activities that span multiple resolution stages; iterative experimentation and adaptive verification; and coordinated changes across artifacts such as datasets, prompts, and model configurations. We also observed challenges related to reproducibility, nondeterministic behavior, and artifact coordination. Building on these findings, we present a vision for AI/ML issue resolution frameworks and discuss research directions and tooling support needed to realize this vision.
