The Weights Can Be Harmful: Pareto Search versus Weighted Search in Multi-objective Search-based Software Engineering
Abstrak
In presence of multiple objectives to be optimized in Search-Based Software Engineering (SBSE), Pareto search has been commonly adopted. It searches for a good approximation of the problem’s Pareto-optimal solutions, from which the stakeholders choose the most preferred solution according to their preferences. However, when clear preferences of the stakeholders (e.g., a set of weights that reflect relative importance between objectives) are available prior to the search, weighted search is believed to be the first choice, since it simplifies the search via converting the original multi-objective problem into a single-objective one and enables the search to focus on what only the stakeholders are interested in. This article questions such a “weighted search first” belief. We show that the weights can, in fact, be harmful to the search process even in the presence of clear preferences. Specifically, we conduct a large-scale empirical study that consists of 38 systems/projects from three representative SBSE problems, together with two types of search budget and nine sets of weights, leading to 604 cases of comparisons. Our key finding is that weighted search reaches a certain level of solution quality by consuming relatively less resources at the early stage of the search; however, Pareto search is significantly better than its weighted counterpart the majority of the time (up to 77% of the cases), as long as we allow a sufficient, but not unrealistic search budget. This is a beneficial result, as it discovers a potentially new “rule-of-thumb” for the SBSE community: Even when clear preferences are available, it is recommended to always consider Pareto search by default for multi-objective SBSE problems, provided that solution quality is more important. Weighted search, in contrast, should only be preferred when the resource/search budget is limited, especially for expensive SBSE problems. This, together with other findings and actionable suggestions in the article, allows us to codify pragmatic and comprehensive guidance on choosing weighted and Pareto search for SBSE under the circumstance that clear preferences are available. All code and data can be accessed at https://github.com/ideas-labo/pareto-vs-weight-for-sbse.
Artikel Ilmiah Terkait
Federica Sarro
1 September 2023
This short paper accompanies the keynote given by Federica Sarro at the 31st IEEE International Requirements Engineering Conference, Hanover, Germany, September 2023.
Yibiao Yang Zuohua Ding Yuming Zhou + 1 lainnya
1 Juni 2022
Software refactoring is one of the most significant practices in software maintenance as the quality of software design tends to deteriorate during software evolution. But, refactoring software is a very challenging task as it requires a holistic view of the entire software system. To this end, recent studies introduced search-based algorithms to facilitate software refactoring. However, they still have the following major limitations: 1) the searched solutions may violate the design principles as their fitness functions do not directly reflect the degree of software’s compliance with design principles; 2) most approaches start the searching process from a completely random initial population, which may lead to unoptimal solutions. In this article, we aim to develop effective search-based refactoring approach to recommend better refactoring activities for developers which can improve the degree of software’s compliance with design principles as well as the software design quality. We propose DEPICTER, a design-principle guided and heuristic-rule constrained software refactoring recommendation approach. In particular, DEPICTER uses non-dominated sorting genetic algorithm (NSGA)-II genetic algorithm and employs design-principle metrics as fitness functions. Besides, DEPICTER leverages heuristic rules to improve the quality of initial population for subsequent generic evolution. Our evaluations, based on four widely used systems, show that DEPICTER is effective for guiding the development of better refactoring models in practice.
M. Gammoudi Ali Ouni Marwa Daaji + 2 lainnya
28 September 2021
Web service composition allows developers to create applications via reusing available services that are interoperable to each other. The process of selecting relevant Web services for a composite service satisfying the developer requirements is commonly acknowledged to be hard and challenging, especially with the exponentially increasing number of available Web services on the Internet. The majority of existing approaches on Web Services Selection are merely based on the Quality of Service (QoS) as a basic criterion to guide the selection process. However, existing approaches tend to ignore the service design quality, which plays a crucial role in discovering, understanding, and reusing service functionalities. Indeed, poorly designed Web service interfaces result in service anti-patterns, which are symptoms of bad design and implementation practices. The existence of anti-pattern instances in Web service interfaces typically complicates their reuse in real-world service-based systems and may lead to several maintenance and evolution problems. To address this issue, we introduce a new approach based on the Multi-Objective and Optimization on the basis of Ratio Analysis method (MOORA) as a multi-criteria decision making (MCDM) method to select Web services based on a combination of their (1) QoS attributes and (2) QoS design. The proposed approach aims to help developers to maintain the soundness and quality of their service composite development processes. We conduct a quantitative and qualitative empirical study to evaluate our approach on a Quality of Web Service dataset. We compare our MOORA-based approach against four commonly used MCDM methods as well as a recent state-of-the-art Web service selection approach. The obtained results show that our approach outperforms state-of-the-art approaches by significantly improving the service selection quality of top-k selected services while providing the best trade-off between both service design quality and desired QoS values. Furthermore, we conducted a qualitative evaluation with developers. The obtained results provide evidence that our approach generates a good trade-off for what developers need regarding both QoS and quality of design. Our selection approach was evaluated as “relevant” from developers point of view, in improving the service selection task with an average score of 3.93, compared to an average of 2.62 for the traditional QoS-based approach.
Jin L. C. Guo Igor Steinmacher M. Nassif + 7 lainnya
2 Maret 2024
Reflecting on the limitations of a study is a crucial part of the research process. In software engineering studies, this reflection is typically conveyed through discussions of study limitations or threats to validity. In current practice, such discussions seldom provide sufficient insight to understand the rationale for decisions taken before and during the study, and their implications. We revisit the practice of discussing study limitations and threats to validity and identify its weaknesses. We propose to refocus this practice of self-reflection to a discussion centered on the notion of trade-offs. We argue that documenting trade-offs allows researchers to clarify how the benefits of their study design decisions outweigh the costs of possible alternatives. We present guidelines for reporting trade-offs in a way that promotes a fair and dispassionate assessment of researchers’ work.
K. Hao Yaochu Jin Qiqi Liu + 1 lainnya
1 Februari 2021
Evolutionary algorithms have been shown to be very successful in solving multi-objective optimization problems (MOPs). However, their performance often deteriorates when solving MOPs with irregular Pareto fronts. To remedy this issue, a large body of research has been performed in recent years and many new algorithms have been proposed. This paper provides a comprehensive survey of the research on MOPs with irregular Pareto fronts. We start with a brief introduction to the basic concepts, followed by a summary of the benchmark test problems with irregular problems, an analysis of the causes of the irregularity, and real-world optimization problems with irregular Pareto fronts. Then, a taxonomy of the existing methodologies for handling irregular problems is given and representative algorithms are reviewed with a discussion of their strengths and weaknesses. Finally, open challenges are pointed out and a few promising future directions are suggested.
Daftar Referensi
0 referensiTidak ada referensi ditemukan.
Artikel yang Mensitasi
0 sitasiTidak ada artikel yang mensitasi.