Weight sensitivity analysis is not sufficient: four further analyst-supplied inputs in multi-criteria decision making for disaster communication selection
Main Article Content
Abstract
Applied multi-criteria decision making (MCDM) studies almost universally report a sensitivity analysis on the criterion weights, and rarely on the other quantities the analyst supplies. This paper contributes a reproducible computational sensitivity-audit protocol and demonstrates it on a disaster communication selection problem used as a testbed: the contribution is the protocol and its findings, not a recommendation of communication media. With fifteen media, six criteria, a non-compensatory mission screen and three aggregations under three hazard weight templates, we perturb five analyst-supplied inputs and evaluate each against the same binary endpoint, whether the recommended leading set changes. Switching the method-plus-preprocessing specification changes it in 67% of method-pair comparisons, of which normalisation alone accounts for a large share. A one-in-twenty disagreement over the performance scores changes it in 28% of draws, against 25% when every weight is randomised by up to 20% in either direction. The screening thresholds and the treatment of the ordinal rating scale as cardinal also move the recommendation. We deliberately do not rank these influences: the perturbation magnitudes are not calibrated against one another, and the ordering of weights against thresholds reverses with the perturbation size, so any ranking would be an artefact of the magnitudes chosen. What the results support is the weaker but still consequential claim that weight sensitivity alone is not an adequate robustness check. A cell-level analysis localises the sensitivity usefully: 53 of 90 score cells cannot alter any recommendation, which makes targeted re-elicitation tractable. We also report descriptively a comparison against media deployed in two Japanese disasters, and explain why the sample cannot support inferential claims.
Article Details
Section
COPYRIGHT
Submission of a manuscript implies: that the work described has not been published before, that it is not under consideration for publication elsewhere; that if and when the manuscript is accepted for publication, the authors agree to automatic transfer of the copyright to the publisher.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
- The journal allows the author(s) to retain publishing rights without restrictions.
- The journal allows the author(s) to hold the copyright without restrictions.
References
A. Mardani, A. Jusoh, and E. K. Zavadskas, “Fuzzy multiple criteria decision-making techniques and applications: two decades review from 1994 to 2014,” Expert Syst. Appl., vol. 42, pp. 4126–4148, 2015.
E. Ilbahar, S. Cebi, and C. Kahraman, “A bibliometric analysis of multi-criteria decision-making techniques in disaster management and transportation in emergencies: towards sustainable solutions,” Sustainability, vol. 17, art. 2644, 2025.
Ministry of Internal Affairs and Communications, “Damage and response concerning the 2024 Noto Peninsula earthquake, report no. 13,” MIC, Tokyo, Japan, Jan. 3, 2024; and White Paper on Information and Communications in Japan 2024, MIC, Tokyo, Japan, 2024.
C. Kahraman, S. C. Onar, and B. Oztaysi, “Fuzzy multicriteria decision-making: a literature review,” Int. J. Comput. Intell. Syst., vol. 8, pp. 637–666, 2015.
F. Kılcı, B. Y. Kara, and B. Bozkaya, “Locating temporary shelter areas after an earthquake: a case for Turkey,” Eur. J. Oper. Res., vol. 243, pp. 323–332, 2015.
E. Triantaphyllou, Multi-Criteria Decision Making Methods: A Comparative Study. Dordrecht, The Netherlands: Kluwer, 2000.
X. Wang and E. Triantaphyllou, “Ranking irregularities when evaluating alternatives by using some ELECTRE methods,” Omega, vol. 36, pp. 45–63, 2008.
R. F. F. Aires and L. Ferreira, “The rank reversal problem in multi-criteria decision making: a literature review,” Pesqui. Oper., vol. 38, pp. 331–362, 2018.
S. Chakraborty and C.-H. Yeh, “A simulation comparison of normalization procedures for TOPSIS,” in Proc. 39th Int. Conf. Computers and Industrial Engineering (CIE39), 2009, pp. 1815–1820.
S. H. Zanakis, A. Solomon, N. Wishart, and S. Dublish, “Multi-attribute decision making: a simulation comparison of select methods,” Eur. J. Oper. Res., vol. 107, pp. 507–529, 1998.
E. K. Zavadskas, Z. Turskis, and S. Kildienė, “State of art surveys of overviews on MCDM/MADM methods,” Technol. Econ. Dev. Econ., vol. 20, pp. 165–179, 2014.
B. Roy, “Classement et choix en présence de points de vue multiples: la méthode ELECTRE,” RIRO, vol. 8, pp. 57–75, 1968.
R. L. Keeney and H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs. New York, NY, USA: Wiley, 1976.
D. Bouyssou and M. Pirlot, “Nontransitive decomposable conjoint measurement,” J. Math. Psychol., vol. 46, pp. 677–703, 2002.
G. Munda, Social Multi-Criteria Evaluation for a Sustainable Economy. Berlin, Germany: Springer, 2008.
J. W. Payne, J. R. Bettman, and E. J. Johnson, The Adaptive Decision Maker. Cambridge, U.K.: Cambridge Univ. Press, 1993.
M. Erdelj, E. Natalizio, K. R. Chowdhury, and I. F. Akyildiz, “Help from the sky: leveraging UAVs for disaster management,” IEEE Pervasive Comput., vol. 16, pp. 24–32, 2017.
T. Sakano et al., “Bringing movable and deployable networks to disaster areas: development and field test of MDRU,” IEEE Netw., vol. 30, pp. 86–91, 2016.
I. del Portillo, B. G. Cameron, and E. F. Crawley, “A technical comparison of three low earth orbit satellite constellation systems to provide global broadband,” Acta Astronaut., vol. 159, pp. 123–135, 2019.
M. M. Kassem, A. Raman, D. Perino, and N. Sastry, “A browser-side view of Starlink connectivity,” in Proc. ACM Internet Measurement Conf., 2022, pp. 151–158.
S. Kassing, D. Bhattacherjee, A. B. Águas, J. E. Saethre, and A. Singla, “Exploring the ‘Internet from space’ with Hypatia,” in Proc. ACM Internet Measurement Conf., 2020, pp. 214–229.
H. Al-Hraishawi, H. Chougrani, S. Kisseleff, E. Lagunas, and S. Chatzinotas, “A survey on nongeostationary satellite systems: the communication perspective,” IEEE Commun. Surveys Tuts., vol. 25, pp. 101–132, 2023.
KDDI Corporation, “Deployment of Starlink to support restoration following the 2024 Noto Peninsula earthquake,” press release, Jan. 2024.
SoftBank Corp., “Behind the scenes: SoftBank’s initiatives to restore communications on the Noto Peninsula,” SoftBank News, Jan. 19, 2024.
C.-L. Hwang and K. Yoon, Multiple Attribute Decision Making: Methods and Applications. Berlin, Germany: Springer, 1981.
J. P. Brans and P. Vincke, “A preference ranking organisation method: the PROMETHEE method for multiple criteria decision-making,” Manage. Sci., vol. 31, pp. 647–656, 1985.
M. Behzadian, R. B. Kazemzadeh, A. Albadvi, and M. Aghdasi, “PROMETHEE: a comprehensive literature review on methodologies and applications,” Eur. J. Oper. Res., vol. 200, pp. 198–215, 2010.
T. L. Saaty, The Analytic Hierarchy Process. New York, NY, USA: McGraw-Hill, 1980.
A. Nonomura, K. Fujisawa, M. Takahashi, H. Matsumoto, and S. Hasegawa, “Analysis of the actions and motivations of a community during the 2017 torrential rains in northern Kyushu, Japan,” Int. J. Environ. Res. Public Health, vol. 17, art. 2424, 2020.
Disaster Prevention Research Institute, Kyoto University, “Disaster survey report on the July 2017 heavy rain in northern Kyushu,” DPRI, Kyoto, Japan, 2017.