Estudo sobre a premiação das áreas de segurança pública no Rio de Janeiro via método multicritério: uma aplicação do método Electre III

Marcio Pereira Basilio; Valdecy Pereira

doi:10.5585/exactaep.v18n1.8725

Authors

Marcio Pereira Basilio Universidade Federal Fluminense https://orcid.org/0000-0002-9453-741X
Valdecy Pereira Universidade Federal Fluminense

DOI:

https://doi.org/10.5585/exactaep.v18n1.8725

Keywords:

MCDA, ELECTRE III, Public security, J-Electre.

Abstract

The present study applied a multi-criteria decision analysis (MCDA) to obtain the ranking of the Integrated Public Security Areas (AISP), for purposes of awarding the Integrated Goal System (SIM). The application of a MCDA had the objective of eliminating the compensatory effect of the criteria, avoiding that success in one criterion compensated for failure in another. The chosen method was ELECTRE III (ELimination Et Choix Traduisant la REalité), because the problem in analysis was adapted to the type (P.γ). As for the purposes, the methodology can be classified as methodological and applied and as for the bibliographical, documentary and laboratory means. The data were obtained in collaboration with the Military Police of the State of Rio de Janeiro, in January 2017, referring to the 12th edition of the SIM award, held in the first half of 2015. The choice of this edition was due to the fact that be the only one in which 11 ISPs have tied in the first place. J-Electre software was used to generate matrices of agreement, disagreement, credibility, partial and final ranking. As a result of the application of the ELECTRE III method, a ranking was obtained without a tie between the first three places. The elimination of the compensatory effect can be perceived by comparing the results obtained by the SIM versus the ELECTRE III, where the order of classification of the units was changed by 94,87%.

Downloads

Download data is not yet available.

Author Biographies

Marcio Pereira Basilio, Universidade Federal Fluminense

Doutorando em Engenharia de Produção, pela Universidade Federal Fluminense (UFF); Mestre em Administração Pública, pela Fundação Getulio Vargas/EBAPE (2007); possuo os seguintes cursos:MBA em Gestão Financeira, Controladoria e Auditoria pela FGV/RJ (2012); MBA em Gestão de Segurança Pública, pela FGV/RJ (2007); Curso Superior de Polícia Integrado, pela Escola Superior de Polícia Militar (2007); Curso de Aperfeiçoamento de Oficiais, pela Escola Superior de Polícia Militar (2000); Pós-graduado em Análise de Sistemas (1999) e Administração de Recursos Humanos (1998); Graduado em Administração de Empresas, pelas Faculdades Reunidas Prof. Nuno Lisboa (1996); o Curso de Formação de Oficiais da Polícia Militar do Estado do Rio de Janeiro (1992). Atualmente estou no posto de coronel e exerço o cargo de Subsecretário Geral da Secretária de Estado da Polícia Militar do Estado do Rio de Janeiro.

Valdecy Pereira, Universidade Federal Fluminense

Possess a degree in Production Engineering (UERJ / 2006), Master in Production Engineering (UFF / 2008), Specialization in Construction Planning and Assembly (UFF / 2008), qualification as Planning and Construction Engineer (PROMINP / 2008) and PhD in Production Engineering (UFF / 2012). Currently Adjunct Professor at the Federal Fluminense University and works mainly on the following topics: Operations Research, Production Planning and Control, Multivariate Data Analysis, Multicriteria Methods, Machine Learning, Deep Machine Learning and Data Mining.

References

Adler, N., Hakkert, A. S., Raviv, T., & Sher, M. (05 de jun de 2014). The Traffic Police Location and Schedule Assignment Problem. Journal of Multi-Criteria Decision Analysis, 21, pp. 315-333. https://dx.doi.org/10.1002/mcda.1522.

Aher, S., Shinde, S., Guha, S., & Majumder, M. (2017). Identification of drought in Dhalai river watershed using MCDM and ANN models. Journal of Earth System Science, 126(2).

Almeida-Dias, J., Figueira, J., & Roy, B. (2012). A multiple criteria sorting method where each category is characterized by several reference actions: The ELECTRE TRI-nC method. European Journal of Operational Research, 217(3), pp. 567-579.

Almeira-Dias, J., Figueira, J., & Roy, B. (2010). ELECTRE TRI-C: A multiple criteria sorting method basead an characteristic reference actions. European Journal of Operational Research, 204(3), pp. 565-580.

Amendola, K. L., Weisburd, D., Hamilton, E. E., Jones, G., & Slipka, M. (2011). An experimental study of compressed work schedules in policing: Advantages and disadvantages of various shift lengths. Journal of Experimental Criminology, 7(4), pp. 407-442. https://dx.doi.org/10.1007/s11292-011-9135-7.

Angelis, A., & Kanavos, P. (2017). Multiple Criteria Decision Analysis (MCDA) for evaluating new medicines in Health Technology Assessment and beyond: The Advance Value Framework. Social Science & Medicine, 188, 137-156.

Angelo, A. C., Saraiva, A. B., Clímaco, J. C., Infante, C. E., & Valle, R. (2017). Life Cycle Assessment and Multi-criteria Decision Analysis: Selection of a strategy for domestic food waste management in Rio de Janeiro. Journal of Cleaner Production, 143, 744-756. https://doi.org/10.1016/j.jclepro.2016.12.049.

Anojkumar, L., Ilangkumaran, M., & Sasirekha, V. (2014). Comparative analysis of MCDM methods for pipe material selection in sugar industry. Expert Systems with Applications, 41(6), 2964-2980. https://doi.org/10.1016/j.eswa.2013.10.028.

Ashari, H. E., & Parsaei, M. (2014). Application of the multi-criteria decision method ELECTRE III for the Weapon selection. Decision Science Letters, 3, pp. 511–522.

Ayag, Z. (2016). A fuzzy ANP approach to evaluate computer simulation packages. International Journal of Business and Systems Research, 10(2-4), 332-345.

Azarnivand, A., & Malekian, A. (2016). Analysis of flood risk management strategies based on a group decision making process via interval-valued intuitionistic fuzzy numbers. Water Resources Management, 30(6), 1903-1921.

Bahadori, M. k., Abolghasemi, K., & Teymourzadeh, E. (2017). Performance evaluation and ranking of selective wards in a military hospital using DEA and promethee method. Journal of Military Medicine, 18(4), pp. 325-334.

Basilio, M. P., de Freitas, J. G., Kämpffe, M. G., & Bordeaux-Rego, R. (2018). Investment portfolio formation via multicriteria decision aid: a brazilian stock market study. Journal of Modelling in Management, 13(2), pp. 394-417. https://doi.org/10.1108/JM2-02-2017-0021.

Basilio, M. P., Pereira, V., & Costa, H. G. (2017). Review of the literature on multicriteria methods applied in the field of public security. Universal Journal of Management, 5(12), pp. 549-562. https://doi.org/10.13189/ujm.2017.051202.

Basilio, M. P., Pereira, V., & Costa, H. G. (2018). Classifying the integrated public safety areas (IPSAs) : a multi-criteria based approach. Journal of Modelling in Management, pp. https://doi.org/10.1108/JM2-01-2018-0001.

Behzadian, M., Kazemzadeh, R. B., Albadvi, A., & Aghdasi, M. (2010). PROMETHEE: A comprehensive literature review on methodologies and applications. European Journal of Operational Research, 200, 198-215.

Behzadian, M., Otaghsara, S. K., Yazdani, M., & Ignatius, J. (2012). A state-of the-art survey of TOPSIS applications. Expert Systems with Applications, 39(17), 13051-13065.

Belton, V., & Stewart, T. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Kluwer Academic Publishers.

Benayoun, R., Roy, B., & Sussman, N. (1966). Manual de Reference du Programme Electre. 25. Paris, France: Note De Synthese et Formaton.

Bernabeu, Y. A. (1980). Manuel D´utilisation du programe ELECTRE III. Cahier du LAMSADE, 35, pp. 1-41.

Blanco, G., Amarilla, R., Martinez, A., Llamosas, C., & Oxilia, V. (2017). Energy transitions and emerging economies: A multi-criteria analysis of policy options for hydropower surplus utilization in Paraguay. Energy Policy, 108, 312-321.

Bouranta, N., Siskos, Y., & Tsotsolas, N. (2015). Measuring police officer and citizen satisfaction: comparative analysis. Policing, 38(4), pp. 705-721. https://dx.doi.org/10.1108/PIJPSM-01-2015-0008.

Camacho-Collados, M., & Liberatore, F. (2015). A Decision Support System for predictive police patrolling. Decision Support Systems, 75, pp. 25-37. https://dx.doi.org/10.1016/j.dss.2015.04.012.

Camacho-Collados, M., Liberatore, F., & Angulo, J. M. (2015). A multi-criteria Police Districting Problem for the efficient and effective design of patrol sector. European Journal of Operational Research, 246(2), pp. 674-684. https://dx.doi.org/10.1016/j.ejor.2015.05.023.

Chen, C. W., Lee, C. C., Tseng, C. P., & Chen, C. H. (2013). Application of GIS for the determination of hazard hotspots after direct transportation linkages between Taiwan and China. Natural Hazards, 66(2), pp. 191-228. https://dx.doi.org/10.1007/s1106.

Chen, F., Wang, J., & Deng, Y. (2015). Road safety risk evaluation by means of improved entropy TOPSIS-RSR. Safety Science, 79, pp. 39-54. https://dx.doi.org/10.1016/j.ssci.2015.05.006.

Cid-López, A., Hornos, M. J., Carrasco, R. A., & Herrera-Viedma, E. (2016). Applying a linguistic multi-criteria decision-making model to the analysis of ICT suppliers' offers. Expert Systems with Applications, 57, 127-138.

De Paula Silva, C. J., Moura, A. C., Paiva, P. C., Ferreira, R. C., Silvestrini, R. A., Vargas, A. M., . . . Ferreira E Ferreira, E. (2015). Maxillofacial injuries as markers of interpersonal violence in belo Horizonte-Brazil: Analysis of the socio-spatial vulnerability of the location of victim's residences. PLoS ONE, 10(8), p. Art. n. e0134577. https://dx.doi.org/10.1371/journal.pone.0134.

Decreto-n.41.930. (25 de junho de 2009). DOERJ. Rio de Janeiro, RJ, Brasil: Casa Civil.

Decreto-n.41.931. (25 de junho de 2009). DOERJ. Rio de Janeiro, RJ, Brasil: Casa Civil.

Decreto-n.42.243. (15 de janeiro de 2010). DOERJ. Rio de Janeiro, RJ, Brasil: Casa Civil.

Decreto-n.42.812. (19 de janeiro de 2011). DOERJ. Rio de Janeiro, RJ, Brasil: Casa Civil.

Decreto-n.43.055. (01 de julho de 2011). DOERJ. Rio de Janeiro, RJ, Brasil: Casa Civil.

Decreto-n.43056. (01 de julho de 2011). DOERJ. Rio de Janeiro, RJ, Brasil: Casa Civil.

Decreto-n.44.348. (23 de agosto de 2013). DOERJ. Rio de Janeiro, RJ, Brasil: Casa Civil.

Decreto-n.45.566. (28 de janeiro de 2016). DOERJ. Rio de Janeiro, RJ, Brasil: Casa Civil.

di Bella, E., Corsi, M., & Leporatti, L. (2014). A Multi-indicator Approach for Smart Security Policy Making. Social Indicators Research, 122(3), pp. 653-675. https://dx.doi.org/10.1007/s11205-014-0714-7.

Diaby, V., Campbell, K., & Goeree, R. (2013). Multi-criteria decision analysis (MCDA) in health care: A bibliometric analysis. Operations Research for Health Care, 2(1-2), 20-24.

Durbach, I. N., & Stewart, T. J. (2012). Modeling uncertainty in multi-criteria decision analysis. European Journal of Operational Research, 223(1), 1-14.

Felice, F., Deldoost, M. H., Faizollahi, M., & Petrillo, A. (2015). Performance Measurement Model for the Supplier Selection Based on AHP. International Journal of Engineering Business Management, 7, 1-13.

Figueira, J., Greco, S., Roy, B., & Slowinski, R. (2010). ELECTRE methods: Main features and recent developments. Em P. M. PARDALOS, D. HEARN, & C. ZOPOUNIDIS, Handbook of multicriteria analysis: aplied optimization (pp. 51-89). Berlin: Springer.

Figueira, J., Greco, S., Roy, B., & Słowiński, R. (2013). An Overview of ELECTRE Methods and their Recent Extensions. Journal of Multi-Criteria Decision Analysis, 20(1-2), 61-85.

Figueira, J., Mousseou, V., & Roy, B. (2005). ELECTRE methods. Em J. FIGUEIRA, S. GRECO, & M. EHRGOTT, Multiple criteria decision analysis: State of the art Survey (pp. 133-153). New York: Springer.

Figueiredo, C. J., & Mota, C. M. (2016). A classification model to evaluate the security level in a city based on GIS-MCDA. Mathematical Problems in Engineering, 2016(Art. nº 3534824), p. https://dx.doi.org/10.1155/2016/3534824.

Franco, C., Bojesen, M., Hougaard, J. L., & Nielsen, K. (2015). A fuzzy approach to a multiple criteria and Geographical Information System for decision support on suitable locations for biogas plants. Applied Energy, 140, 304-315.

Giannoulis, C., & Ishizaka, A. (2010). A Web-based Decision Support System with ELECTRE III for a Personalised Ranking of British Universities. Decision Support Systems, 48(3), pp. 488–497.

Gigović, L., Pamučar, D., Bajić, Z., & Drobnjak, S. (2017). Application of GIS-Interval Rough AHP Methodology for Flood Hazard Mapping in Urban Areas. Water, 9(6), 360.

Govindan, K., & Jepsen, M. B. (2016). ELECTRE: A comprehensive literature review on methodologies and applications. European Journal of Operational Research, 250, pp. 1-29.

Gupta, M., Chandra, B., & Gupta, M. P. (2014). A framework of intelligent decision support system for Indian police. Journal of Enterprise Information Management, 27(5), pp. 512-540. https://dx.doi.org/10.1108/JEIM-10-2012-0073.

Gurgel, A. M., & Mota, C. M. (2013). A multicriteria prioritization model to support public safety planning. Pesquisa Operacional, 33(2), pp. 251-267. https://dx.doi.org/10.1590/S0101-74382013000200007.

Hodgett, R. E. (2016). Comparison of multi-criteria decision-making methods for equipment selection. The International Journal of Advanced Manufacturing Technology, 85(5), pp. 1145–1157. doi:10.1007/s00170-015-7993-2.

Hosseinzade, Z., Pagsuyoin, S. A., Ponnambalam, K., & Monem, M. J. (2017). Decision-making in irrigation networks: Selecting appropriate canal structures using multi-attribute decision analysis. Science of The Total Environment, 601-602, 177-185.

Hu, S.-K., Lu, M.-T., & Tzeng, G.-H. (2015). Improving Mobile Commerce Adoption Using a New Hybrid Fuzzy MADM Model. International Journal of Fuzzy Systems, 17(8), 399-413.

Huang, I. B., Keisler, J., & Linkov, I. (2011). Multi-criteria decision analysis in environmental sciences: Ten years of applications and trends. Science of the Total Environment, 409(19), 3578-3594.

Huda, S., Sarno, R., & Ahmad, T. (2015). Fuzzy MADM approach for rating of process-based fraud. Journal of ICT Research and Applications, 9(2), 111-128.

Hwang, C. L., & Yoon, K. (1981). Multiple attribute decision making: methods and applications, a state-of-the-art survey. New York: Spring-Verlay.

Infante, C. E., Mendonça, F. M., & Valle, R. A. (2014). Análise de robustez com o método Electre III: o caso da região de Campo das Vertentes em Minas Gerais. Gestão & Produção, 21(2), pp. 245-255.

Ishizaka , A., & Pereira , V. E. (2016). Portraying an employee performance management system based on multi-criteria decision analysis and visual techniques. International Journal of Manpower, 37(4), 628-659. https://doi-org.ez24.periodicos.capes.gov.br/10.1108/IJM-07-2014-0149.

Janstrup, K., Kaplan, S., Barfod, M., & Prato, C. (2017). Evaluating the police service quality for handling traffic crash reporting: A combined MCDA and LCA approach. Policing: An International Journal of Police Strategies & Management, 40(2), 410-425. https://doi-org.ez24.periodicos.capes.gov.br/10.1108/PIJPSM-03-2016-0032.

Jozi, S. A., Shoshtory, M. T., & Zadeh, A. R. (2015). Environmental Risk Assessment of Dams in Construction Phase Using a Multi-Criteria Decision-Making (MCDM) Method. Human and Ecological Risk Assessment: An International Journal, 21(1), 1-16.

Kahraman, C., Onar, S. C., & Oztaysi, B. (2015). Fuzzy Multicriteria Decision-Making: A Literature Review. International Journal of Computational Intelligence Systems, 8(4), 637-666.

Karlson, M., Karlsson, C. S., Mörtberg, U., Olofsson, B., & Balfors, B. (2016). Design and evaluation of railway corridors based on spatial ecological and geological criteria. Transportation Research Part D: Transport and Environment, 46, 207-228.

Karvetski, C. W., Lambert, J. H., & Linkovz, I. (2011). Scenario and multiple criteria decision analysis for energy and environmental security of military and industrial installations. Integrated Environmental Assessment and Management, 7(2), pp. 228-236. https://dx.doi.org/10.1002/ieam.137.

Kim, Y., & Chung, E. S. (2013). Assessing climate change vulnerability with group multi-criteria decision making approaches. Climatic Change, 121(2), 301-315.

Lau, H. C., Ho, G. T., Zhao, Y., & Hon, W. T. (2010). Optimizing patrol force deployment using a genetic algorithm. Expert Systems with Applications, 37(12), pp. 8148-8154. https://dx.doi.org/10.1016/j.eswa.2010.05.073.

Leoneti, A. B. (2016). Considerations regarding the choice of ranking multiple criteria decision making methods. Pesquisa Operacional, 36(2), pp. 259-277. https://dx.doi.org/10.1590/0101-7438.2016.036.02.0259.

Liberatore, F., & Camacho-Collados, M. (2016). A Comparison of Local Search Methods for the Multicriteria Police Districting Problem on Graph. Mathematical Problems in Engineering, 2016(Art. nº 3690474), p. https://dx.doi.org/10.1155/2016/3690474.

Linkov, I., & Seager, T. P. (2011). Coupling multi-criteria decision analysis, life-cycle assessment, and risk assessment for emerging threats. Environmental Science and Technology, 45(12), 5068-5074.

Linkov, I., Bates, M. E., Canis, L. J., Seager, T. P., & Keisler, J. M. (2011). A decision-directed approach for prioritizing research into the impact of nanomaterials on the environment and human health. Nature Nanotechnology, 6(12), 784-787.

Linkov, I., Trump, B., Jin, D., Mazurczak, M., & Schreurs, M. (2014). A decision-analytic approach to predict state regulation of hydraulic fracturing. Environmental Sciences Europe, 26(1).

Lopez, J. C., Noriega, J. J., & Chavira, D. A. (2017). A Multi-Criteria Approach to Rank the Municipalities of the States of Mexico by its Marginalization Level: The Case of Jalisco. International Journal of Information Technology & Decision Making, 16(2), pp. 473-513. http://dx.doi.org/10.1142/S0219622017500080.

Maimoun, M., Madani, K., & Reinhart, D. (2016). Multi-level multi-criteria analysis of alternative fuels for waste collection vehicles in the United States. Science of the Total Environment, 550, 349-361.

Manning, M., Smith, C., & Homel, R. (2013). Overview of: "Valuing developmental crime prevention". Criminology and Public Policy, 12(2), pp. https://dx.doi.org/10.1111/1745-9133.12023.

Mendoza, G. A., & Martins, H. (15 de July de 2006). Multi-criteria decision analysis in natural resource management: A critical review of methods and new modelling paradigms. Forest Ecology and Management, 230(1-3), pp. 1-22. https://doi.org/10.1016/j.foreco.2006.03.023.

Mignelli, C., Pomarico, S., & Peila, D. (2013). Use of multi-criteria model to compare devices for the protection of roads against Rockfall . Environmental and Engineering Geoscience, 19(3), pp. 289-302. https://dx.doi.org/10.2113/gseegeosci.19.3.289.

Montazer, G. A., Saremi, H. Q., & Ramezani, M. (2009). Design a new mixed expert decision aiding system using fuzzy ELECTRE III method for vendor selection. Expert Systems with Applications, 36(8), pp. 10837-10847.

Mousavi-Nasab, S. H., & Sotoudeh-Anvari, A. (2017). A comprehensive MCDM-based approach using TOPSIS, COPRAS and DEA as an auxiliary tool for material selection problems. Materials and Design, 121, pp. 237-253.

Mühlbacher, A. C., & Kaczynski, A. (2016). Making good decisions in healthcare with multi-criteria decision analysis: The use, current research and future development of MCDA. Applied Health Economics and Health Policy, 14(1), 29-40.

Nutt, D. J., King, L. A., & Phillips, L. D. (2010). Drug harms in the UK: A multicriteria decision analysis. The Lancet, 376(9752), pp. 1558-1565. https://dx.doi.org/10.1016/S0140-6736(10)61462-6.

Özdemir, Ü., & Güneroğlu, A. (2015). Strategic approach model for investigating the cause of maritime accidents [Stratejik bir model yaklaşimi ile deniz kaza sebeplerinin araştirilmasi]. Promet - Traffic – Traffico, 27(2), pp. 113-123. https://dx.doi.org/10.7307/ptt.v27i2.1461.

Pape, T. (2016). Prioritising data items for business analytics: Framework and application to human resources. European Journal of Operational Research, 252(2), 687-698.

Pereira, V., Costa, H. G., & Nepomuceno, L. D. (2016). J-Electre (V.1.0) (software). Niteroí, RJ, Brazil.

Potić, I., Golić, R., & Joksimović, T. (2016). Analysis of insolation potential of Knjaževac Municipality (Serbia) using multi-criteria approach. Renewable and Sustainable Energy Reviews, 56, 235-245.

Resolução n° 305. (13 de janeiro de 2010). DOERJ. Rio de Janeiro, Rj, Brasil: Secretaria de Estado de Segurança.

Roy, B. (1968). Classement et choix en présence de points de vue multiples (la méthode ELECTRE). RIRO, 8, pp. 57-75.

Roy, B. (1978). ELECTRE III: Un algorithme de classement fondé sur une représentation floue des préférences en présence de critères multiples. Cahiers du CERO, 20, pp. 3-24.

Roy, B. (1990). Decision-aid and decision-making. European Journal of Operational Research, 45(2-3), pp. 324-331.

Roy, B., & Bouyssou, D. (1993). Aide multicritère à La décision: méthodes et cãs. Paris: Ed. Economica.

Roy, B., & Slowinski, R. (Mars de 2013). Questions guiding the choice of a multicriteria decision aiding method. Cahiers du LAMSADE, Nº 337.

Roy, B., & Bertier, P. (1971). La methode ELECTRE II: Une methode de classement en presence de critteres multiples. SEMA (Metra International), Direction Scientifique, Note de Travail No. 142, Paris.

Roy, B., & Hugonnard, J. C. (1982). Classement des prolongements de métro en banlieue parisienne(Présentation d´une méthode multicritère originale). Cahiers du CERO, 23, 153-171.

Roy, B., & Slowinski, R. (1984). ELECTRE IS - Aspects méthodologiques et guide d´utilisation. Cahiers du LAMSADE, 30, 125p.

Sakthivel, G., Ilangkumaran, M., & Gaikwad, A. (2014). A hybrid multi-criteria decision modeling approach for the best biodiesel blend selection based on ANP-TOPSIS analysis. Ain Shams Engineering Journal, 6(1), 239-256.

Schmitz, S., McCullagh, L., Adams, R., Barry, M., & Walsh, C. (2016). Identifying and revealing the importance of decision-making criteria for health technology assessment: A retrospective analysis of reimbursement recommendations in Ireland. PharmacoEconomics, 34(9), 925-937.

Scholten, L., Maurer, M., & Lienert, J. (2017). Comparing multi-criteria decision analysis and integrated assessment to support long-term water supply planning. PLoS ONE, 12(5), https://doi.org/10.1371/journal.pone.0176663.

Shen, J., Lu, H., Zhang, Y., Song, X., & He, L. (2016). Vulnerability assessment of urban ecosystems driven by water resources, human health and atmospheric environment. Journal of Hydrology, 536, 457-470.

Shen, K.-Y., Hu, S.-K., & Tzeng, G.-H. (2017). Financial modeling and improvement planning for the life insurance industry by using a rough knowledge based hybrid MCDM model. Information Sciences, 375, 296-313. https://doi.org/10.1016/j.ins.2016.09.055.

Shmelev, S. E., & Van Den Bergh, J. (2016). Optimal diversity of renewable energy alternatives under multiple criteria: An application to the UK. Renewable and Sustainable Energy Reviews, 60, 679-691.

Singh, B., Grover, S., & Singh, V. (2017). An empirical study of benchmarking evaluation using MCDM in service industries. Managerial Auditing Journal, 32(2), 111-147. https://doi-org.ez24.periodicos.capes.gov.br/10.1108/MAJ-11-2015-1274.

Trestian, R., Ormond, O., & Muntean, G.-M. (2015). Performance evaluation of MADM-based methods for network selection in a multimedia wireless environment. Wireless Networks, 21(5), 1745-1763.

Triantaphyllou, E. (2000). Multi-criteria decision making methods: a comparative study. Dordrecht: Springer Science+Business Media.

Vaidya, O., & Kumar, S. (2006). Analytic hierarchy process: An overview of applications. European Journal of Operational Research, 169(1), 1-29. https://doi.org/10.1016/j.ejor.2004.04.028.

Vallée, D., & Zielniewicz, P. (1994). ELECTRE III-IV version 3.x: aspects méthodologiques (tome 1). Université Paris-Dauphine, Laboratoire d´Analyse et Modélisation de Systèmes pour l´Aide à la Décision. Paris: Document du LAMSADE Nº 85.

Velasquez, M., & Hester, P. T. (2013). An analysis of multi-criteria decision making methods. International Journal of Operations Research, 10(2), 56-66.

Vincke, P. (1992). Multicriteria decision-aid. John Wiley & Sons.

Wallenius, J., Dyer, J., Fishburn, P., Steuer, R., Zionts, S., & Deb, K. (2008). Multiple Criteria Decision Making, Multiattribute Utility Theory: Recent Accomplishments and What Lies Ahead. Management Science, 54(7), 1336-1349.

Wan, S. P., Xu, G. l., & Dong, J. Y. (2017). Supplier selection using ANP and ELECTRE II in interval 2-tuple linguistic environment. Information Sciences, 385-386, pp. 19-38.

Wang, J. J., Jing, Y. Y., Zhang, C. F., & Zhao, J. H. (2009). Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renewable and Sustainable Energy Reviews, 13(9), 2263-2278.

Wang, P., Zhu, Z., & Wang, Y. (2016). A novel hybrid MCDM model combining the SAW, TOPSIS and GRA methods based on experimental design. Information Sciences, 345, 27-45.

Wang, X., & Triantaphyllou, E. (2006). Ranking irregularities when evaluating alternatives by using some multi-criteria decision analysis methods. Em A. Badiru (Ed.), Handbook of Industrial and Systems Engineeting (pp. 27-12). FL, U.S.A: CRC Press, Taylor & Francis Group.

Yu, W. (1992). Electre Tri — aspects méthodologiques et guide d’utilisation. Document du LAMSADE, 74. Paris: Université de Paris-Dauphine.

Zong, F., & Wang, L. (2017). Evaluation of university scientific research ability based on the output of sci-tech papers: A D-AHP approach. PLoS ONE, 12(2), e0171437. https://doi.org/10.1371/journal.pone.0171437.

Zyoud, S. H., & Fuchs-Hanusch, D. (2017). A bibliometric-based survey on AHP and TOPSIS techniques. Expert System with Applications, 78, 158-181. http://dx.doi.org/10.1016/j.eswa.2017.02.016.