运行时增强的LLM用于ML笔记本的崩溃检测与诊断
CRANE-LLM: Runtime-Augmented LLMs for Crash Prediction and Diagnosis in ML Notebooks
AI总结：
CRANE-LLM通过整合运行时信息提升ML笔记本崩溃检测与诊断的准确性与F1分数。
摘要：
Jupyter笔记本因其支持交互式和迭代实验而被广泛用于机器学习（ML）开发。然而，ML笔记本极易出现错误，崩溃是最具破坏性的现象之一。尽管其实际重要性，系统化的方法在ML笔记本中进行崩溃检测和诊断仍然鲜有探索。我们提出了CRANE-LLM，一种新颖的方法，通过从笔记本内核状态中提取结构化运行时信息来增强大型语言模型（LLMs），以在执行目标单元之前检测和诊断崩溃。给定之前执行的单元和目标单元，CRANE-LLM结合静态代码上下文与运行时信息，包括对象类型、张量形状和数据属性，以预测目标单元是否会崩溃（检测）并解释其根本原因（诊断）。我们在JunoBench上评估CRANE-LLM，这是一个包含222个ML笔记本的基准，涵盖111对崩溃和相应非崩溃笔记本，跨越多种ML库和崩溃根本原因。在三种最先进的LLM（Gemini、Qwen和GPT-5）上，运行时信息在崩溃检测和诊断的准确率上提高了7-10个百分点，在F1分数上提高了8-11个百分点，对诊断的改进更大。改进在不同ML库、崩溃原因和LLM之间有所不同，并依赖于互补类别的运行时信息的整合。
英文摘要：
Jupyter notebooks have become popular for early machine learning (ML) development, enabling interactive and iterative experimentation. However, ML notebooks are prone to bugs, among which crashes are the most disruptive. Despite their practical importance, crash prediction and diagnosis in ML notebooks remain largely unexplored. We present CRANE-LLM, a runtime-augmented source code analysis approach that provides large language models (LLMs) with structured runtime information extracted from the notebook kernel, together with source code, to predict and diagnose crashes in a target cell before executing it. We evaluate CRANE-LLM on JunoBench, a benchmark of 111 Kaggle ML notebooks containing crashes across multiple ML libraries and crash types. Across three state-of-the-art LLMs (Gemini, Qwen, and GPT-5), our results show that runtime information significantly improves crash prediction and diagnosis performance by 7-10 percentage points in accuracy and 8-11 in F1-score, compared to using source code alone. The improvements are more pronounced when diagnosis is required, indicating that runtime context is especially valuable for reasoning about crash causes than merely predicting their presence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
