面向机器学习 Notebook 理解能力的静态分析驱动增强方法
Static Analysis Driven Enhancements for Comprehension in Machine Learning Notebooks
AI总结：
本文提出 HeaderGen 工具，通过增强 PyCG 调用图分析并引入流敏感性和外部库返回类型解析，自动为机器学习 notebook 代码单元生成分类标题，从而提升 notebook 的可读性与理解效率，实验显示其在调用图分析和标题生成上均具有较高准确率。
摘要：
Jupyter notebook 使开发者能够将代码片段与富文本和内联可视化交错编排。数据科学家将 Jupyter notebook 作为创建和共享基于机器学习的解决方案的事实标准，这些解决方案主要使用 Python 编写。然而，近期研究表明，公共平台上可用的大部分 Jupyter notebook 缺乏文档说明，并且缺少叙事结构。这降低了这些 notebook 的可读性。为了解决这一不足，本文提出了 HeaderGen，这是一种新颖的基于工具的方法，它根据机器学习操作的分类法，自动为代码单元标注分类 markdown 标题，并根据该分类法对函数调用进行分类和显示。为了实现这一功能，HeaderGen 增强了 PyCG 中现有的调用图分析。为了提高精度，HeaderGen 扩展了 PyCG 的分析能力，使其支持处理外部库代码和流敏感性。前者通过促进函数返回类型的解析来实现。在来自 Kaggle 的 15 个真实世界 Jupyter notebook 上的评估表明，HeaderGen 底层的调用图分析具有很高的准确性（95.6% 的精确率和 95.3% 的召回率）。这是因为 HeaderGen 能够解析外部库的返回类型，而现有的类型推断工具如 pytype（由 Google 开发）、pyright（由 Microsoft 开发）和 Jedi 在这方面存在不足。标题生成的精确率为 85.7%，召回率为 92.8%。在一项用户研究中，HeaderGen 帮助参与者更快地完成理解和导航任务。为了进一步评估工具的类型推断能力，我们引入了 TypeEvalPy，这是一个用于评估类型推断工具的框架，包含一个包含 154 个代码片段和 845 个类型注解的微基准测试。我们对四个工具的比较分析表明，HeaderGen 在与真实值的精确匹配方面优于其他工具。
英文摘要：
Jupyter notebooks enable developers to interleave code snippets with rich-text and in-line visualizations. Data scientists use Jupyter notebook as the de-facto standard for creating and sharing machine-learning based solutions, primarily written in Python. Recent studies have demonstrated, however, that a large portion of Jupyter notebooks available on public platforms are undocumented and lacks a narrative structure. This reduces the readability of these notebooks. To address this shortcoming, this paper presents HeaderGen, a novel tool-based approach that automatically annotates code cells with categorical markdown headers based on a taxonomy of ML operations, and classifies and displays function calls according to this taxonomy. For this functionality to be realized, HeaderGen enhances an existing call graph analysis in PyCG. To improve precision, HeaderGen extends PyCG's analysis with support for handling external library code and flow-sensitivity. The former is realized by facilitating the resolution of function return-types. The evaluation on 15 real-world Jupyter notebooks from Kaggle shows that HeaderGen's underlying call graph analysis yields high accuracy (95.6% precision and 95.3% recall). This is because HeaderGen can resolve return-types of external libraries where existing type inference tools such as pytype (by Google), pyright (by Microsoft), and Jedi fall short. The header generation has a precision of 85.7% and a recall rate of 92.8%. In a user study, HeaderGen helps participants finish comprehension and navigation tasks faster. To further evaluate the type inference capability of tools, we introduce TypeEvalPy, a framework for evaluating type inference tools with a micro-benchmark containing 154 code snippets and 845 type annotations. Our comparative analysis on four tools revealed that HeaderGen outperforms other tools in exact matches with the ground truth.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
