如何在基于Pareto搜索的软件工程中评估解决方案？批判性审查和方法指导
How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance
摘要：
随着现代需求的增加，在许多软件工程（SE）场景中同时考虑多个目标/标准的趋势越来越明显。这样的多目标优化场景伴随着一个重要的问题-如何评估优化算法的结果，其通常是一组不可比的解决方案（即，彼此之间是帕累托非支配的）。这个问题对于SE社区来说可能是一个挑战，特别是对于基于搜索的SE（SBSE）的从业者来说。一方面，多目标优化对SE/SBSE研究人员来说仍然是相对较新的，他们可能无法为他们的问题确定正确的评估方法。另一方面，简单地遵循一般多目标优化问题的评估方法可能不适合特定的SE问题，特别是当问题的性质或决策者的偏好显式/隐式可用时。这在文献中得到了很好的回应，各种不适当/不充分的选择和不准确/误导性的使用评价方法。在本文中，我们首先进行了系统的和批判性的审查质量评价多目标优化SBSE。我们调查了2009年至2019年期间发表的717篇论文，这些论文来自7个知识库的36个地点，并选择了95项突出的研究，通过这些研究，我们确定了该领域中五个重要但被忽视的问题。然后，我们进行了深入的分析，质量评价指标/方法和一般情况下，SBSE，其中，连同所确定的问题，使我们能够编纂的方法指南，选择和使用评估方法在不同的SBSE场景。
英文摘要：
With modern requirements, there is an increasing tendency of considering multiple objectives/criteria simultaneously in many Software Engineering (SE) scenarios. Such a multi-objective optimization scenario comes with an important issue -- how to evaluate the outcome of optimization algorithms, which typically is a set of incomparable solutions (i.e., being Pareto non-dominated to each other). This issue can be challenging for the SE community, particularly for practitioners of Search-Based SE (SBSE). On one hand, multi-objective optimization could still be relatively new to SE/SBSE researchers, who may not be able to identify the right evaluation methods for their problems. On the other hand, simply following the evaluation methods for general multi-objective optimization problems may not be appropriate for specific SE problems, especially when the problem nature or decision maker's preferences are explicitly/implicitly available. This has been well echoed in the literature by various inappropriate/inadequate selection and inaccurate/misleading use of evaluation methods. In this paper, we first carry out a systematic and critical review of quality evaluation for multi-objective optimization in SBSE. We survey 717 papers published between 2009 and 2019 from 36 venues in seven repositories, and select 95 prominent studies, through which we identify five important but overlooked issues in the area. We then conduct an in-depth analysis of quality evaluation indicators/methods and general situations in SBSE, which, together with the identified issues, enables us to codify a methodological guidance for selecting and using evaluation methods in different SBSE scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
