arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1909.07249cs.SE
收藏

专家系统辅助文献综述的评价与案例研究

Assessing Expert System-Assisted Literature Reviews With a Case Study

Zhe Yu, Jeffrey C. Carver, Gregg Rothermel, Tim Menzies

摘要:

鉴于软件工程的大量出版物,需要经常进行文献综述,以保持特定领域的最新工作。在文献综述中,一项繁琐的工作是在数千个不相关的搜索结果中找到相关的研究。从理论上讲,专家系统可以帮助找到相关的工作,但这些系统主要是在模拟测试,而不是在应用到实际的文献综述。因此,很少有研究人员相信这样的专家系统。因此,使用一个现实的案例研究,本文评估如何以及我们的国家的最先进的专家系统可以帮助文献综述。评估的文献综述旨在识别自动化UI测试的测试用例优先级技术,特别是来自IEEE Xplore的8,349篇论文。该语料库进行了研究与专家系统,其中包括一个增量更新的人在环主动学习工具。使用该专家系统,在三个小时内,我们找到了242个相关的论文,我们确定了12个技术代表的国家的最先进的测试用例优先级时,源代码信息不可用。这些结果,然后验证了其他六个研究生手动探索相同的语料库。如果没有专家系统,这项任务将需要53个小时,并将找到27份额外的文件。也就是说,我们的专家系统实现了90%的召回与6%的人力成本相比,传统的手工方法。值得注意的是,专家系统和手动方法都确定了相同的12种最先进的测试用例优先级技术。也就是说,专家系统遗漏的27篇论文不会改变文献综述的结论。因此,如果这一结果普遍化,它赞同使用我们的专家系统,以协助文献综述。

英文摘要:

Given the large number of publications in software engineering, frequent literature reviews are required to keep current on work in specific areas. One tedious work in literature reviews is to find relevant studies amongst thousands of non-relevant search results. In theory, expert systems can assist in finding relevant work but those systems have primarily been tested in simulations rather than in application to actual literature reviews. Hence, few researchers have faith in such expert systems. Accordingly, using a realistic case study, this paper assesses how well our state-of-the-art expert system can help with literature reviews. The assessed literature review aimed at identifying test case prioritization techniques for automated UI testing, specifically from 8,349 papers on IEEE Xplore. This corpus was studied with an expert system that incorporates an incrementally updated human-in-the-loop active learning tool. Using that expert system, in three hours, we found 242 relevant papers from which we identified 12 techniques representing the state-of-the-art in test case prioritization when source code information is not available. These results were then validated by six other graduate students manually exploring the same corpus. Without the expert system, this task would have required 53 hours and would have found 27 additional papers. That is, our expert system achieved 90% recall with 6% of the human effort cost when compared to a conventional manual method. Significantly, the same 12 state-of-the-art test case prioritization techniques were identified by both the expert system and the manual method. That is, the 27 papers missed by the expert system would not have changed the conclusion of the literature review. Hence, if this result generalizes, it endorses the use of our expert system to assist in literature reviews.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑