Enhancing quality in online surveys: uncovering foundational themes and strategies
DOI:
https://doi.org/10.5585/remark.v23i4.25692Palabras clave:
Methodology, Online questionnaire, Survey quality, Data qualityResumen
Purpose: This study aimed to identify the foundational themes and strategies to enhance response quality in online surveys.
Method: The term "online survey" and its variations were employed as broad keywords for the article selection process to identify methodological and empirical articles about online survey quality. The selected database was characterized using bibliometric techniques, the foundational themes were identified through co-citation analysis, and recommended strategies were determined using bibliographic coupling analysis.
Findings: The foundational themes in online survey literature encompass Device, Mode of Administration, Question Design, Careless Responses, Response Rate, Paradata, Statistical Adjustment, Incentives, and Household Survey. Device and Mode of Administration emphasize the description and comparison of online collection methods with traditional approaches and explore the use of different devices. The remaining themes investigate strategies aimed at enhancing online survey response, focusing on specific strategies, quality indicators, or participant behaviors.
Originality/Value: This study serves as a valuable guideline for survey researchers. To the authors’ knowledge, this is the first review using co-citation analysis to identify the main strategies for improving response quality in online survey research.
Theoretical/Methodological contributions: This research contributes across many disciplines by uncovering the main strategies to improve response quality in online survey research and providing guidance for survey researchers.
Social/Managerial contributions: By highlighting the importance of survey protocols and the potential biases and errors associated with unplanned research, the findings offer practical insights for social and managerial contexts.
Descargas
Citas
Al-Salom, P., & Miller, C. J. (2019). The Problem with Online Data Collection: Predicting Invalid Responding in Undergraduate Samples. Current Psychology, 38(5), 1258–1264. https://doi.org/10.1007/s12144-017-9674-9
Alwin, D. F., & Krosnick, J. A. (1991). The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes. Sociological Methods & Research, 20(1). https://doi.org/10.1177/0049124191020001005
Bethlehem, J. (2010). Selection bias in web surveys. International Statistical Review, 78(2), 161-188. https://doi.org/10.1111/j.1751-5823.2010.00112.x
Biemer, P. P. (2010). Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly, 74(5), 817–848. https://doi.org/10.1093/poq/nfq058
Bosnjak, M., Dannwolf, T., Enderle, T., Schaurer, I., Struminskaya, B., Tanner, A., & Weyandt, K. W. (2018). Establishing an Open Probability-Based Mixed-Mode Panel of the General Population in Germany: The GESIS Panel. Social Science Computer Review, 36(1). https://doi.org/10.1177/0894439317697949
Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27(3), 281–291. https://doi.org/10.1093/pubmed/fdi031
Bybee, S., Cloyes, K., Baucom, B., Supiano, K., Mooney, K., & Ellington, L. (2022). Bots and nots: safeguarding online survey research with underrepresented and diverse populations. Psychology & Sexuality, 13(4), 901-911. https://doi.org/10.1080/19419899.2021.1936617
Callegaro, M. (2013). Paradata in Web Surveys. In Kreuter, F. (Ed.), Improving surveys with paradata: Analytic uses of process information (pp. 259–279). John Wiley & Sons. https://doi.org/10.1002/9781118596869.ch11
Cornesse, C., & Bosnjak, M. (2018, April). Is there an association between survey characteristics and representativeness? A meta-analysis. Survey Research Methods. 12(1), 1-13. https://doi.org/10.18148/srm/2018.v12i1.7205
Couper, M. P. (2011). The future of modes of data collection. Public Opinion Quarterly, 75(5), 889-908. https://doi.org/10.1093/poq/nfr046
Couper, M. P., Kapteyn, A., Schonlau, M., & Winter, J. (2007). Noncoverage and nonresponse in an Internet survey. Social Science Research, 36(1), 131-148. https://doi.org/10.1016/j.ssresearch.2005.10.002
Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19(2), 146-162. https://doi.org/10.1177/089443930101900202
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-9. https://doi.org/10.1016/j.jesp.2015.07.006
Daikeler, J., Bosnjak, M., & Manfreda, K. L. (2020). Web versus other survey modes: An updated and extended meta-analysis comparing response rates. Journal of Survey Statistics and Methodology, 8(3), 513–539. https://doi.org/10.1093/jssam/smz008
De Man, J., Campbell, L., Tabana, H., & Wouters, E. (2021). The pandemic of online research in times of COVID-19. BMJ Open, 11(2). https://doi.org/10.1136/bmjopen-2020-043866
DeCastellarnau, A. (2018). A classification of response scale characteristics that affect data quality: a literature review. Quality & quantity, 52(4), 1523-1559. https://doi.org/10.1007/s11135-017-0533-4
DeLeeuw, E. D. (2018, August). Mixed-mode: Past, present, and future. Survey Research Methods. 12(2), 75-89. https://doi.org/10.18148/srm/2018.v12i2.7402
Deutskens, E., de Ruyter, K., Wetzels, M., & Oosterveld, P. (2004). Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study. Marketing Letters, 15(1), 21–36. https://doi.org/10.1023/B:MARK.0000021968.86465.00
Dever, J. A., Rafferty, A., & Valliant, R. (2008). Internet surveys: Can statistical adjustments eliminate coverage bias? Survey Research Methods, 2(2), 47–60. https://doi.org/10.18148/srm/2008.v2i2.128
Dillman, D. A., & Christian, L. M. (2005). Survey Mode as a Source of Instability in Responses across Surveys. Field Methods, 17(1), 30-52. https://doi.org/10.1177/1525822X04269550
Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., & Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of business research, 133, 285-296. https://doi.org/10.1016/j.jbusres.2021.04.070
Evans, J. R., & Mathur, A. (2018). The value of online surveys: a look back and a look ahead. Internet Research, 28(4), 854–887. https://doi.org/10.1108/IntR-03-2018-0089
Ferri-García, R., & Rueda, M. D. M. (2020). Propensity score adjustment using machine learning classification algorithms to control selection bias in online surveys. PloS one, 15(4), e0231500. https://doi.org/10.1371/journal.pone.0231500
Freire, O., Senise, D. D. S. V., dos Reis, W. B., & Ono, H. S. (2017). # Step@ by# Step: recomendações para o desenvolvimento de pesquisa online de alta qualidade. Revista Gestão & Tecnologia, 17(3), 36-62. https://doi.org/10.20397/2177-6652/2017.v17i3.1281
Galesic, M., Tourangeau, R., Couper, M. P., & Conrad, F. G. (2008). Eye-tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72(5), 892–913. https://doi.org/10.1093/poq/nfn059
Göritz, A. S. (2006). Incentives in web studies: Methodological issues and a review. International Journal of Internet Science, 1(1), 58-70. https://jwolf-ra.wdfiles.com/local--files/incentives/goritz_2006.pdf
Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646–675. https://doi.org/10.1093/poq/nfl033
Groves, R. M., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public Opinion Quarterly, 74(5), 849–879. https://doi.org/10.1093/poq/nfq065
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68, 2–31. https://doi.org/10.1093/poq/nfh002
Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64, 299–308. https://doi.org/10.1086/317990
Hair, J. F. (2009). Multivariate data analysis (7th ed.). Prentice Hall.
Hulland, J., Baumgartner, H., & Smith, K. M. (2018). Marketing survey research best practices: evidence and recommendations from a review of JAMS articles. Journal of the Academy of Marketing Science, 46(1), 92–108. https://doi.org/10.1007/s11747-017-0532-y
Iannelli, L., Giglietto, F., Rossi, L., & Zurovac, E. (2020). Facebook digital traces for survey research: Assessing the efficiency and effectiveness of a Facebook Ad–based procedure for recruiting online survey respondents in niche and difficult-to-reach populations. Social Science Computer Review, 38(4), 462-476. https://doi.org/10.1177/0894439318816638
Johnson, J. A. (2005). Ascertaining the validity of individual protocols from Web-based personality inventories. Journal of Research in Personality, 39(1), 103-129. https://doi.org/10.1016/j.jrp.2004.09.009
Keusch, F. (2015). Why do people participate in Web surveys? Applying survey participation theory to Internet survey data collection. Management Review Quarterly, 65(3), 183–216. https://doi.org/10.1007/s11301-014-0111-y
Keusch, F., & Zhang, C. (2017). A review of issues in gamified surveys. Social Science Computer Review, 35(2), 147-166. https://doi.org/10.1177/0894439315608451
Keusch, F., Bähr, S., Haas, G. C., Kreuter, F., & Trappmann, M. (2023). Coverage error in data collection combining mobile surveys with passive measurement using apps: Data from a German national survey. Sociological Methods & Research, 52(2), 841-878. https://doi.org/10.1177/0049124120914924
Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. T., Michael Hanemann, W., Kopp, R. J., … Conaway, M. (2002). The Impact Of “No Opinion” Response Options On Data Quality Non-Attitude Reduction Or An Invitation To Satisfice? Public Opinion Quarterly, 66(3), 371–403. https://doi.org/10.1086/341394
Lee, S. (2006). Propensity score adjustment as a weighting scheme for volunteer panel web surveys. Journal of Official Statistics, 22(2), 329-349. https://www.proquest.com/docview/1266792612
Leeuw, E. D. De. (2005). To Mix or Not to Mix Data Collection Modes in Surveys. Journal of Official Statistics, 21(2), 233-255. https://www.proquest.com/docview/1266791766
Lehdonvirta, V., Oksanen, A., Räsänen, P., & Blank, G. (2021). Social Media, Web, and Panel Surveys: Using Non-Probability Samples in Social and Policy Research. Policy and Internet, 13(1), 134–155. https://doi.org/10.1002/poi3.238
Leiner, D. J. (2019). Too fast, too straight, too weird: Non-reactive indicators for meaningless data in internet surveys. Survey Research Methods, 13(3), 229–248. https://doi.org/10.18148/srm/2019.v13i3.7403
Liebe, U., Glenk, K., von Meyer-Höfer, M., & Spiller, A. (2019). A web survey application of real choice experiments. Journal of choice modelling, 33, 100150. https://doi.org/10.1016/j.jocm.2018.07.003
Lugtig, P., & Toepoel, V. (2016). The use of PCs, smartphones, and tablets in a probability-based panel survey: Effects on survey measurement error. Social science computer review, 34(1), 78-94. https://doi.org/10.1177/0894439315574248
Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48(1), 61-83. https://doi.org/10.1016/j.jrp.2013.09.008
Marcus, B., Bosnjak, M., Lindner, S., Pilischenko, S., & Schütz, A. (2007). Compensating for low topic interest and long surveys: A field experiment on nonresponse in web surveys. Social Science Computer Review, 25(3), 372-383. https://doi.org/10.1177/0894439307297606
Matias, J., Kalamara, E., Mathis, F., Skarupova, K., Noor, A., Singleton, N., … Grp, E. W. S. D. (2019). The use of multi-national web surveys for comparative analysis: Lessons from the European Web Survey on Drugs. International Journal of Drug Policy, 73, 235–244. https://doi.org/10.1016/j.drugpo.2019.03.014
Mavletova, A., & Couper, M. P. (2014). Mobile web survey design: Scrolling versus paging, sms versus e-mail invitations. Journal of Survey Statistics and Methodology, 2(4), 498–518. https://doi.org/10.1093/jssam/smu015
McCain, K. W. (1990). Mapping authors in intellectual space: A technical overview. Journal of the American Society for Information Science, 41(6), 433-443. https://doi.org/10.1002/(SICI)1097-4571(199009)41:6<433::AID-ASI11>3.0.CO;2-Q
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085
Olson, K., Smyth, J. D., Horwitz, R., Keeter, S., Lesser, V., Marken, S., … Wagner, J. (2021). Transitions From Telephone Surveys To Self-Administered And Mixed-Mode Surveys: Aapor Task Force Report. Journal of Survey Statistics and Methodology, 9(3), 381–411. https://doi.org/10.1093/jssam/smz062
Palaniappan, K., & Kum, I. Y. S. (2019). Underlying Causes behind Research Study Participants’ Careless and Biased Responses in the Field of Sciences. Current Psychology, 38(6), 1737–1747. https://doi.org/10.1007/s12144-017-9733-2
Paul, J., Lim, W. M., O’Cass, A., Hao, A. W., & Bresciani, S. (2021). Scientific procedures and rationales for systematic literature reviews (SPAR-4-SLR). International Journal of Consumer Studies, 45(4), O1-O16. https://doi.org/10.1111/ijcs.12695
Peytchev, A., & Hill, C. A. (2010). Experiments in mobile web survey design: Similarities to other modes and unique considerations. Social Science Computer Review, 28(3), 319-335. https://doi.org/10.1177/0894439309353037
Pickert, R. (2023). Survey Response Rates Are Down Since Covid. That’s Worrying for US Economic Data. Bloomberg. https://www.bloomberg.com/news/articles/2023-02-15/us-economic-data-threatened-by-survey-non-responses
Quevedo-Silva, F., Almeida Santos, E. B., Brandão, M. M., & Vils, L. (2016). Estudo Bibliométrico: Orientações sobre sua Aplicação. Revista Brasileira de Marketing, 15(2), 246–262. https://doi.org/10.5585/remark.v15i2.3274
Revilla, M., & Ochoa, C. (2017). Ideal and maximum length for a web survey. International Journal of Market Research, 59(5), 557-565. https://doi.org/10.2501/IJMR-2017-039
Roßmann, J., Gummer, T., & Silber, H. (2018). Mitigating satisficing in cognitively demanding grid questions: Evidence from two web-based experiments. Journal of Survey Statistics and Methodology, 6(3), 376-400. https://doi.org/10.1093/jssam/smx020
Rouse, S. V. (2019). Reliability of MTurk data from masters and workers. Journal of Individual Differences. 41(1), 30–36. https://doi.org/10.1027/1614-0001/a000300
Sammut, R., Griscti, O., & Norman, I. J. (2021). Strategies to improve response rates to web surveys: a literature review. International Journal of Nursing Studies, 123, 104058. https://doi.org/10.1016/j.ijnurstu.2021.104058
Saris, W. E., Revilla, M., Krosnick, J. A., & Shaeffer, E. M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61–79.. https://doi.org/10.18148/srm/2010.v4i1.2682
Sauer, P. C., & Seuring, S. (2023). How to conduct systematic literature reviews in management research: a guide in 6 steps and 14 decisions. Review of Managerial Science, 17, 1899–1933. https://doi.org/10.1007/s11846-023-00668-3
Sauermann, H., & Roach, M. (2013). Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features. Research Policy, 42(1), 273-286. https://doi.org/10.1016/j.respol.2012.05.003
Shamon, H., & Berning, C. C. (2020). Attention Check Items and Instructions in Online Surveys with Incentivized and Non-Incentivized Samples: Boon or Bane for Data Quality?. Survey Research Methods, 14(1), 55-77. https://doi.org/10.18148/srm/2020.v14i1.7374
Singer, E., & Ye, C. (2013). The Use and Effects of Incentives in Surveys. Annals of the American Academy of Political and Social Science, 645(1), 112-141. https://doi.org/10.1177/0002716212458082
Smith, S. M., Roster, C. A., Golden, L. L., & Albaum, G. S. (2016). A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples. Journal of Business Research, 69(8), 3139–3148. https://doi.org/10.1016/j.jbusres.2015.12.002
Stoet, G. (2017). PsyToolkit: A Novel Web-Based Method for Running Online Questionnaires and Reaction-Time Experiments. Teaching of Psychology, 44(1), 24–31. https://doi.org/10.1177/0098628316677643
Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020). Got bots? Practical recommendations to protect online survey data from bot attacks. The Quantitative Methods for Psychology, 16(5), 472-481. https://doi.org/10.20982/tqmp.16.5.p472
Struminskaya, B., Weyandt, K., & Bosnjak, M. (2015). The Effects of Questionnaire Completion Using Mobile Devices on Data Quality. Evidence from a Probability-based General Population Panel. Methods, Data, Analyses, 9(2). https://doi.org/10.12758/mda.2015.014
Tourangeau, R. (2003). Cognitive aspects of survey measurement and mismeasurement. International Journal of Public Opinion Research, 15(1), 3–7. https://doi.org/10.1093/ijpor/15.1.3
Tourangeau, R., Couper, M. P., & Conrad, F. (2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368–393. https://doi.org/10.1093/poq/nfh035
Tourangeau, R., Couper, M. P., & Conrad, F. (2007). Color, labels, and interpretive heuristics for response scales. Public Opinion Quarterly, 71(1), 91–112. https://doi.org/10.1093/poq/nfl046
Walzenbach, S. (2019). Hiding Sensitive Topics by Design? An Experiment on the Reduction of Social Desirability Bias in Factorial Surveys. Survey Research Methods, 13(1), 103–121. https://doi.org/10.18148/srm/2019.v1i1.7243
Weber, S. (2021). A step-by-step procedure to implement discrete choice experiments in Qualtrics. Social Science Computer Review, 39(5), 903-921. https://doi.org/10.1177/0894439319885317
Wells, T., Bailey, J. T., & Link, M. W. (2014). Comparison of Smartphone and Online Computer Survey Administration. Social Science Computer Review, 32(2), 238-255. https://doi.org/10.1177/0894439313505829
Zupic, I., & Čater, T. (2015). Bibliometric Methods in Management and Organization. Organizational Research Methods, 18(3), 429-472. https://doi.org/10.1177/1094428114562629
Zwarun, L., & Hall, A. (2014). What’s going on? Age, distraction, and multitasking during online survey taking. Computers in Human Behavior, 41, 236-244. https://doi.org/10.1016/j.chb.2014.09.041
Descargas
Publicado
Cómo citar
Número
Sección
Licencia
Derechos de autor 2024 Fernanda Sayuri Yoda, Otávio Bandeira de Lamônica Freire
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial-CompartirIgual 4.0.
