arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.07696cs.SE
收藏

面向机器学习视觉洞察到分析断言的自动翻译

Towards Automatic Translation of Machine Learning Visual Insights to Analytical Assertions

Arumoy Shome, Luis Cruz, Arie van Deursen

AI总结:

本文提出将机器学习可视化中的视觉属性自动翻译为 Python 断言,基于从 GitHub 挖掘的 269 对可视化-断言对构建分类体系,并比较传统 NLP4Code 模型与大型语言模型的效果,以增强 ML 系统验证。

摘要:

我们提出了开发一种自动化工具的愿景,该工具能够将在机器学习(ML)可视化中观察到的视觉属性翻译为 Python 断言。该工具旨在简化 ML 开发周期中手动验证这些可视化的过程,这一点至关重要,因为现实世界的数据和假设在部署后经常会发生变化。在先前的一项研究中,我们从 Github 挖掘了 54,070 个 Jupyter notebook,并创建了一个包含 269 对语义相关的可视化-断言(VA)对的目录。在此目录的基础上,我们提出构建一个分类体系,根据 ML 验证任务来组织这些 VA 对。输入特征空间包含从 Jupyter notebook 中挖掘出的丰富信息来源——可视化、Python 源代码以及相关的 markdown 文本。我们将使用既定的机器翻译指标,比较各种 AI 模型(包括传统的 NLP4Code 模型和现代大型语言模型)的有效性,并通过一项有人类参与者参与的定性研究进行评估。本文还计划解决从 Kaggle 中获取额外配对来扩展现有 VA 对数据集的挑战,并将该工具的有效性与 ChatGPT 等商业生成式 AI 模型进行比较。这项研究不仅为 ML 系统验证领域做出了贡献,还探索了利用 AI 自动化和增强 ML 中软件工程实践的新方法。

英文摘要:

We present our vision for developing an automated tool capable of translating visual properties observed in Machine Learning (ML) visualisations into Python assertions. The tool aims to streamline the process of manually verifying these visualisations in the ML development cycle, which is critical as real-world data and assumptions often change post-deployment. In a prior study, we mined $54,070$ Jupyter notebooks from Github and created a catalogue of $269$ semantically related visualisation-assertion (VA) pairs. Building on this catalogue, we propose to build a taxonomy that organises the VA pairs based on ML verification tasks. The input feature space comprises of a rich source of information mined from the Jupyter notebooks -- visualisations, Python source code, and associated markdown text. The effectiveness of various AI models, including traditional NLP4Code models and modern Large Language Models, will be compared using established machine translation metrics and evaluated through a qualitative study with human participants. The paper also plans to address the challenge of extending the existing VA pair dataset with additional pairs from Kaggle and to compare the tool's effectiveness with commercial generative AI models like ChatGPT. This research not only contributes to the field of ML system validation but also explores novel ways to leverage AI for automating and enhancing software engineering practices in ML.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑