The Significance of Parameters’ Optimization in Fair Benchmarking of Software Defects’ Prediction Performances
Main Article Content
Abstract
Software engineering research in general and software defects’ prediction research in particular are facing serious challenges to their reliability and validity. The major reason is that many of the published research outcomes contradict each other. This phenomenon is mainly caused by the lack of research standards as it exists in many well-established scientific and engineering disciplines. The scope of this paper is to focus on fair benchmarking of the defects’ prediction models. By experimenting three prediction algorithms, we found that the quality of the resultant predictions would significantly fluctuate as parameters’ values changed. Therefore, any published research results not based on optimized prediction algorithms methods can cause inaccurate and misleading benchmarking and recommendations. Thus, we propose optimizing parameters as an essential research standard to conduct reliable and valid benchmarking. We believe if this standard were approved by interested software quality practitioners and research communities, it will present a vital role in soothing the severity of this phenomenon. The three prediction algorithms we used in our analysis were Support Vector Machine SVM, Multilayer Perceptron MLP, and Naïve Bayesian NB. We used KNIME as a data mining platform to design and run all optimization loops on open source Eclipse 2.0 data set.
Â
Keywords: Parameters’ optimization, defects prediction, data mining, benchmarking, performance quality, SVM, MLP, NB, KNIME, Eclipse, machine learning
Downloads
Article Details
COPYRIGHT
Submission of a manuscript implies: that the work described has not been published before, that it is not under consideration for publication elsewhere; that if and when the manuscript is accepted for publication, the authors agree to automatic transfer of the copyright to the publisher.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
- The journal allows the author(s) to retain publishing rights without restrictions.
- The journal allows the author(s) to hold the copyright without restrictions.