This paper proposes a new challenge problem for software analytics. In the process we shall call "software review", a panel of SMEs (subject matter experts) review examples of software behavior to recommend how to improve that's software's operation. SME time is usually extremely limited so, ideally, this panel can complete this optimization task after looking at just a small number of very informative, examples.
To support this review process, we explore methods that train a predictive model to guess if some oracle will like/dislike the next example. Such a predictive model can work with the SMEs to guide them in their exploration of all the examples. Also, after the panelists leave, that model can be used as an oracle in place of the panel (to handle new examples, while the panelists are busy, elsewhere).
In 31 case studies (ranging from from high-level decisions about software processes to low-level decisions about how to configure video encoding software), we show that such predictive models can be built using as few as 12 to 30 labels. To the best of our knowledge, this paper's success with only a handful of examples (and no large language model) is unprecedented.
In accordance with the principles of open science, we offer all our code and data at https://github.com/timm/ez/tree/Stable-EMSE-paper so that others can repeat/refute/improve these results.
SharkTrack: an accurate, generalisable software for streamlining shark and ray underwater video analysis
Imperial College London(帝国理工学院);Shark Measurements(鲨鱼测量公司);Virginia Tech(弗吉尼亚理工大学);James Cook University(詹姆斯库克大学);Florida International University(佛罗里达国际大学);University of Exeter(埃克塞特大学);Anguilla National Trust(安圭拉国家信托基金);Government of Anguilla(安圭拉政府);Bangor University(班戈大学);University of Leeds(利兹大学)
How Far Does the Predictive Decision Impact the Software Project? The Cost, Service Time, and Failure Analysis from a Cross-Project Defect Prediction Model
Breaking New Ground in Software Defect Prediction: Introducing Practical and Actionable Metrics with Superior Predictive Power for Enhanced Decision-Making
Predicting Abandonment of Open Source Software Projects with An Integrated Feature Framework
School of Computer Science, Peking University(北京大学计算机学院);Key Laboratory of High Confidence Software Technologies, Ministry of Education(教育部高可信软件技术重点实验室);National University of Defense Technology(国防科技大学);National Key Laboratory of Parallel and Distributed Computing(并行与分布式处理国家重点实验室)
A Model using agile methodologies for defining metrics to be used by the Public Sector in Brazil to set remuneration for outsourced software development