arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.12920cs.SE
收藏

简化软件评审:基于少量样本的高效预测建模

Streamlining Software Reviews: Efficient Predictive Modeling with Minimal Examples

Tim Menzies, Andre Lustosa

AI总结:

针对软件评审中主题专家时间有限的问题,提出训练预测模型辅助评审,31项案例研究显示仅需12-30个标注样本即可构建有效模型,且未使用大语言模型,相关代码数据已开源。

摘要:

本文提出了一个软件分析领域的新挑战问题。在我们称为“软件评审”的过程中,由SME(主题专家)组成的评审小组会审查软件行为样本,以给出改进软件运行方式的建议。SME的时间通常极其有限,因此理想情况下,该评审小组只需查看少量极具信息量的样本,就能完成这项优化任务。\n 为支持这一评审流程,我们探索了训练预测模型的方法,该模型可预测某个预言机(oracle)是否会喜欢/不喜欢下一个样本。这种预测模型可与SME协同工作,引导他们完成对所有样本的探索。此外,在评审专家离开后,该模型还可代替评审小组充当预言机(在评审专家忙于其他事务时处理新样本)。\n 在31项案例研究(涵盖从软件流程的高层决策到视频编码软件配置方式的底层决策等多种场景)中,我们证明仅需12到30个标注样本就能构建这类预测模型。据我们所知,本文仅用少量样本(且未使用大语言模型)就取得成功的成果是前所未有的。\n 遵循开放科学原则,我们在https://github.com/timm/ez/tree/Stable-EMSE-paper 提供了所有代码和数据,以便其他研究者重复、验证或改进这些结果。

英文摘要:

This paper proposes a new challenge problem for software analytics. In the process we shall call "software review", a panel of SMEs (subject matter experts) review examples of software behavior to recommend how to improve that's software's operation. SME time is usually extremely limited so, ideally, this panel can complete this optimization task after looking at just a small number of very informative, examples. To support this review process, we explore methods that train a predictive model to guess if some oracle will like/dislike the next example. Such a predictive model can work with the SMEs to guide them in their exploration of all the examples. Also, after the panelists leave, that model can be used as an oracle in place of the panel (to handle new examples, while the panelists are busy, elsewhere). In 31 case studies (ranging from from high-level decisions about software processes to low-level decisions about how to configure video encoding software), we show that such predictive models can be built using as few as 12 to 30 labels. To the best of our knowledge, this paper's success with only a handful of examples (and no large language model) is unprecedented. In accordance with the principles of open science, we offer all our code and data at https://github.com/timm/ez/tree/Stable-EMSE-paper so that others can repeat/refute/improve these results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑