超越准确性:经验数据的ROI驱动数据分析
Beyond Accuracy: ROI-driven Data Analytics of Empirical Data
摘要:
这份愿景文件表明,在执行数据分析时,考虑投资回报率(ROI)至关重要。关于“需要多少分析”的决策?很难回答投资回报率可以指导决策支持什么?,怎么做?多少钱?分析一个给定的问题。方法:提出的概念框架进行了验证,通过两个实证研究,重点是在Mozilla Firefox项目的需求依赖提取。这两个案例研究是(i)针对朴素贝叶斯和随机森林机器学习器对微调BERT进行二进制依赖分类的评估,以及(ii)针对REQUIRES依赖提取的被动学习(随机采样)的主动学习。对于这两种情况,他们的分析投资(成本)进行了估计,并预测可实现的收益从DA,以确定一个盈亏平衡点的调查。结果:对于第一项研究,微调BERT表现优于随机森林,前提是超过40%的训练数据可用。对于第二个,与基线(基于随机采样的RF分类器)相比,主动学习在更少的迭代内实现了更高的F1准确度和更高的ROI。在这两项研究中,估计有多少分析可能会为投资的努力带来回报?是由盈亏平衡点来表示的。结论:决策的深度和广度的DA的经验数据不应该仅仅基于准确性的措施。由于ROI驱动的数据分析提供了一个简单而有效的方向,可以发现何时停止进一步调查,同时考虑各种类型分析的成本和价值,它有助于避免过度分析经验数据。
英文摘要:
This vision paper demonstrates that it is crucial to consider Return-on-Investment (ROI) when performing Data Analytics. Decisions on "How much analytics is needed"? are hard to answer. ROI could guide for decision support on the What?, How?, and How Much? analytics for a given problem. Method: The proposed conceptual framework is validated through two empirical studies that focus on requirements dependencies extraction in the Mozilla Firefox project. The two case studies are (i) Evaluation of fine-tuned BERT against Naive Bayes and Random Forest machine learners for binary dependency classification and (ii) Active Learning against passive Learning (random sampling) for REQUIRES dependency extraction. For both the cases, their analysis investment (cost) is estimated, and the achievable benefit from DA is predicted, to determine a break-even point of the investigation. Results: For the first study, fine-tuned BERT performed superior to the Random Forest, provided that more than 40% of training data is available. For the second, Active Learning achieved higher F1 accuracy within fewer iterations and higher ROI compared to Baseline (Random sampling based RF classifier). In both the studies, estimate on, How much analysis likely would pay off for the invested efforts?, was indicated by the break-even point. Conclusions: Decisions for the depth and breadth of DA of empirical data should not be made solely based on the accuracy measures. Since ROI-driven Data Analytics provides a simple yet effective direction to discover when to stop further investigation while considering the cost and value of the various types of analysis, it helps to avoid over-analyzing empirical data.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。