Enhancing quality in online surveys: uncovering foundational themes and strategies

Fernanda Sayuri Yoda; Otávio Bandeira de Lamônica Freire

doi:10.5585/remark.v23i4.25692

Authors

Fernanda Sayuri Yoda University of São Paulo https://orcid.org/0000-0001-8259-1315
Otávio Bandeira de Lamônica Freire Universidade de São Paulo https://orcid.org/0000-0002-9008-4273

DOI:

https://doi.org/10.5585/remark.v23i4.25692

Keywords:

Methodology, Online questionnaire, Survey quality, Data quality

Abstract

Purpose: This study aimed to identify the foundational themes and strategies to enhance response quality in online surveys.

Method: The term "online survey" and its variations were employed as broad keywords for the article selection process to identify methodological and empirical articles about online survey quality. The selected database was characterized using bibliometric techniques, the foundational themes were identified through co-citation analysis, and recommended strategies were determined using bibliographic coupling analysis.

Findings: The foundational themes in online survey literature encompass Device, Mode of Administration, Question Design, Careless Responses, Response Rate, Paradata, Statistical Adjustment, Incentives, and Household Survey. Device and Mode of Administration emphasize the description and comparison of online collection methods with traditional approaches and explore the use of different devices. The remaining themes investigate strategies aimed at enhancing online survey response, focusing on specific strategies, quality indicators, or participant behaviors.

Originality/Value: This study serves as a valuable guideline for survey researchers. To the authors’ knowledge, this is the first review using co-citation analysis to identify the main strategies for improving response quality in online survey research.

Theoretical/Methodological contributions: This research contributes across many disciplines by uncovering the main strategies to improve response quality in online survey research and providing guidance for survey researchers.

Social/Managerial contributions: By highlighting the importance of survey protocols and the potential biases and errors associated with unplanned research, the findings offer practical insights for social and managerial contexts.

Downloads

Download data is not yet available.

Author Biographies

Fernanda Sayuri Yoda, University of São Paulo

Business & Management Ph.D. student - PPGA-FEA/USP

Marketing Graduate Program Adjunct professor - ESPM

Marketing Graduate Program Adjunct professor - Grupo Primo

Marketing Graduate Program Adjunct professor - FIA

Otávio Bandeira de Lamônica Freire, Universidade de São Paulo

Business & Management Post-Graduate Program Professor - PPGA-FEA/USP Tourism Management Post-Graduate Program Professor - PPGTUR-EACH/USP Undergraduate Marketing Professor - EACH/USP Associate Editor for Marketing - RAUSP Management Journal Associate Editor - RMR Retail Management Review Founding-Partner & Head of Science - ILUMEO Data Science Company Coordinator - Branding, Relationships, Analytics, Network, Data and Strategy Research Center - B.R.A.N.D.S/USP Vice-Coordinator - Tourism Economics and Management Research Center NEAT/USP

References

Al-Salom, P., & Miller, C. J. (2019). The Problem with Online Data Collection: Predicting Invalid Responding in Undergraduate Samples. Current Psychology, 38(5), 1258–1264. https://doi.org/10.1007/s12144-017-9674-9

Alwin, D. F., & Krosnick, J. A. (1991). The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes. Sociological Methods & Research, 20(1). https://doi.org/10.1177/0049124191020001005

Bethlehem, J. (2010). Selection bias in web surveys. International Statistical Review, 78(2), 161-188. https://doi.org/10.1111/j.1751-5823.2010.00112.x

Biemer, P. P. (2010). Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly, 74(5), 817–848. https://doi.org/10.1093/poq/nfq058

Bosnjak, M., Dannwolf, T., Enderle, T., Schaurer, I., Struminskaya, B., Tanner, A., & Weyandt, K. W. (2018). Establishing an Open Probability-Based Mixed-Mode Panel of the General Population in Germany: The GESIS Panel. Social Science Computer Review, 36(1). https://doi.org/10.1177/0894439317697949

Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27(3), 281–291. https://doi.org/10.1093/pubmed/fdi031

Bybee, S., Cloyes, K., Baucom, B., Supiano, K., Mooney, K., & Ellington, L. (2022). Bots and nots: safeguarding online survey research with underrepresented and diverse populations. Psychology & Sexuality, 13(4), 901-911. https://doi.org/10.1080/19419899.2021.1936617

Callegaro, M. (2013). Paradata in Web Surveys. In Kreuter, F. (Ed.), Improving surveys with paradata: Analytic uses of process information (pp. 259–279). John Wiley & Sons. https://doi.org/10.1002/9781118596869.ch11

Cornesse, C., & Bosnjak, M. (2018, April). Is there an association between survey characteristics and representativeness? A meta-analysis. Survey Research Methods. 12(1), 1-13. https://doi.org/10.18148/srm/2018.v12i1.7205

Couper, M. P. (2011). The future of modes of data collection. Public Opinion Quarterly, 75(5), 889-908. https://doi.org/10.1093/poq/nfr046

Couper, M. P., Kapteyn, A., Schonlau, M., & Winter, J. (2007). Noncoverage and nonresponse in an Internet survey. Social Science Research, 36(1), 131-148. https://doi.org/10.1016/j.ssresearch.2005.10.002

Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19(2), 146-162. https://doi.org/10.1177/089443930101900202

Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-9. https://doi.org/10.1016/j.jesp.2015.07.006

Daikeler, J., Bosnjak, M., & Manfreda, K. L. (2020). Web versus other survey modes: An updated and extended meta-analysis comparing response rates. Journal of Survey Statistics and Methodology, 8(3), 513–539. https://doi.org/10.1093/jssam/smz008

De Man, J., Campbell, L., Tabana, H., & Wouters, E. (2021). The pandemic of online research in times of COVID-19. BMJ Open, 11(2). https://doi.org/10.1136/bmjopen-2020-043866

DeCastellarnau, A. (2018). A classification of response scale characteristics that affect data quality: a literature review. Quality & quantity, 52(4), 1523-1559. https://doi.org/10.1007/s11135-017-0533-4

DeLeeuw, E. D. (2018, August). Mixed-mode: Past, present, and future. Survey Research Methods. 12(2), 75-89. https://doi.org/10.18148/srm/2018.v12i2.7402

Deutskens, E., de Ruyter, K., Wetzels, M., & Oosterveld, P. (2004). Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study. Marketing Letters, 15(1), 21–36. https://doi.org/10.1023/B:MARK.0000021968.86465.00

Dever, J. A., Rafferty, A., & Valliant, R. (2008). Internet surveys: Can statistical adjustments eliminate coverage bias? Survey Research Methods, 2(2), 47–60. https://doi.org/10.18148/srm/2008.v2i2.128

Dillman, D. A., & Christian, L. M. (2005). Survey Mode as a Source of Instability in Responses across Surveys. Field Methods, 17(1), 30-52. https://doi.org/10.1177/1525822X04269550

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., & Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of business research, 133, 285-296. https://doi.org/10.1016/j.jbusres.2021.04.070

Evans, J. R., & Mathur, A. (2018). The value of online surveys: a look back and a look ahead. Internet Research, 28(4), 854–887. https://doi.org/10.1108/IntR-03-2018-0089

Ferri-García, R., & Rueda, M. D. M. (2020). Propensity score adjustment using machine learning classification algorithms to control selection bias in online surveys. PloS one, 15(4), e0231500. https://doi.org/10.1371/journal.pone.0231500

Freire, O., Senise, D. D. S. V., dos Reis, W. B., & Ono, H. S. (2017). # Step@ by# Step: recomendações para o desenvolvimento de pesquisa online de alta qualidade. Revista Gestão & Tecnologia, 17(3), 36-62. https://doi.org/10.20397/2177-6652/2017.v17i3.1281

Galesic, M., Tourangeau, R., Couper, M. P., & Conrad, F. G. (2008). Eye-tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72(5), 892–913. https://doi.org/10.1093/poq/nfn059

Göritz, A. S. (2006). Incentives in web studies: Methodological issues and a review. International Journal of Internet Science, 1(1), 58-70. https://jwolf-ra.wdfiles.com/local--files/incentives/goritz_2006.pdf

Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646–675. https://doi.org/10.1093/poq/nfl033

Groves, R. M., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public Opinion Quarterly, 74(5), 849–879. https://doi.org/10.1093/poq/nfq065

Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68, 2–31. https://doi.org/10.1093/poq/nfh002

Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64, 299–308. https://doi.org/10.1086/317990

Hair, J. F. (2009). Multivariate data analysis (7th ed.). Prentice Hall.

Hulland, J., Baumgartner, H., & Smith, K. M. (2018). Marketing survey research best practices: evidence and recommendations from a review of JAMS articles. Journal of the Academy of Marketing Science, 46(1), 92–108. https://doi.org/10.1007/s11747-017-0532-y

Iannelli, L., Giglietto, F., Rossi, L., & Zurovac, E. (2020). Facebook digital traces for survey research: Assessing the efficiency and effectiveness of a Facebook Ad–based procedure for recruiting online survey respondents in niche and difficult-to-reach populations. Social Science Computer Review, 38(4), 462-476. https://doi.org/10.1177/0894439318816638

Johnson, J. A. (2005). Ascertaining the validity of individual protocols from Web-based personality inventories. Journal of Research in Personality, 39(1), 103-129. https://doi.org/10.1016/j.jrp.2004.09.009

Keusch, F. (2015). Why do people participate in Web surveys? Applying survey participation theory to Internet survey data collection. Management Review Quarterly, 65(3), 183–216. https://doi.org/10.1007/s11301-014-0111-y

Keusch, F., & Zhang, C. (2017). A review of issues in gamified surveys. Social Science Computer Review, 35(2), 147-166. https://doi.org/10.1177/0894439315608451

Keusch, F., Bähr, S., Haas, G. C., Kreuter, F., & Trappmann, M. (2023). Coverage error in data collection combining mobile surveys with passive measurement using apps: Data from a German national survey. Sociological Methods & Research, 52(2), 841-878. https://doi.org/10.1177/0049124120914924

Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. T., Michael Hanemann, W., Kopp, R. J., … Conaway, M. (2002). The Impact Of “No Opinion” Response Options On Data Quality Non-Attitude Reduction Or An Invitation To Satisfice? Public Opinion Quarterly, 66(3), 371–403. https://doi.org/10.1086/341394

Lee, S. (2006). Propensity score adjustment as a weighting scheme for volunteer panel web surveys. Journal of Official Statistics, 22(2), 329-349. https://www.proquest.com/docview/1266792612

Leeuw, E. D. De. (2005). To Mix or Not to Mix Data Collection Modes in Surveys. Journal of Official Statistics, 21(2), 233-255. https://www.proquest.com/docview/1266791766

Lehdonvirta, V., Oksanen, A., Räsänen, P., & Blank, G. (2021). Social Media, Web, and Panel Surveys: Using Non-Probability Samples in Social and Policy Research. Policy and Internet, 13(1), 134–155. https://doi.org/10.1002/poi3.238

Leiner, D. J. (2019). Too fast, too straight, too weird: Non-reactive indicators for meaningless data in internet surveys. Survey Research Methods, 13(3), 229–248. https://doi.org/10.18148/srm/2019.v13i3.7403

Liebe, U., Glenk, K., von Meyer-Höfer, M., & Spiller, A. (2019). A web survey application of real choice experiments. Journal of choice modelling, 33, 100150. https://doi.org/10.1016/j.jocm.2018.07.003

Lugtig, P., & Toepoel, V. (2016). The use of PCs, smartphones, and tablets in a probability-based panel survey: Effects on survey measurement error. Social science computer review, 34(1), 78-94. https://doi.org/10.1177/0894439315574248

Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48(1), 61-83. https://doi.org/10.1016/j.jrp.2013.09.008

Marcus, B., Bosnjak, M., Lindner, S., Pilischenko, S., & Schütz, A. (2007). Compensating for low topic interest and long surveys: A field experiment on nonresponse in web surveys. Social Science Computer Review, 25(3), 372-383. https://doi.org/10.1177/0894439307297606

Matias, J., Kalamara, E., Mathis, F., Skarupova, K., Noor, A., Singleton, N., … Grp, E. W. S. D. (2019). The use of multi-national web surveys for comparative analysis: Lessons from the European Web Survey on Drugs. International Journal of Drug Policy, 73, 235–244. https://doi.org/10.1016/j.drugpo.2019.03.014

Mavletova, A., & Couper, M. P. (2014). Mobile web survey design: Scrolling versus paging, sms versus e-mail invitations. Journal of Survey Statistics and Methodology, 2(4), 498–518. https://doi.org/10.1093/jssam/smu015

McCain, K. W. (1990). Mapping authors in intellectual space: A technical overview. Journal of the American Society for Information Science, 41(6), 433-443. https://doi.org/10.1002/(SICI)1097-4571(199009)41:6<433::AID-ASI11>3.0.CO;2-Q

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085

Olson, K., Smyth, J. D., Horwitz, R., Keeter, S., Lesser, V., Marken, S., … Wagner, J. (2021). Transitions From Telephone Surveys To Self-Administered And Mixed-Mode Surveys: Aapor Task Force Report. Journal of Survey Statistics and Methodology, 9(3), 381–411. https://doi.org/10.1093/jssam/smz062

Palaniappan, K., & Kum, I. Y. S. (2019). Underlying Causes behind Research Study Participants’ Careless and Biased Responses in the Field of Sciences. Current Psychology, 38(6), 1737–1747. https://doi.org/10.1007/s12144-017-9733-2

Paul, J., Lim, W. M., O’Cass, A., Hao, A. W., & Bresciani, S. (2021). Scientific procedures and rationales for systematic literature reviews (SPAR-4-SLR). International Journal of Consumer Studies, 45(4), O1-O16. https://doi.org/10.1111/ijcs.12695

Peytchev, A., & Hill, C. A. (2010). Experiments in mobile web survey design: Similarities to other modes and unique considerations. Social Science Computer Review, 28(3), 319-335. https://doi.org/10.1177/0894439309353037

Pickert, R. (2023). Survey Response Rates Are Down Since Covid. That’s Worrying for US Economic Data. Bloomberg. https://www.bloomberg.com/news/articles/2023-02-15/us-economic-data-threatened-by-survey-non-responses

Quevedo-Silva, F., Almeida Santos, E. B., Brandão, M. M., & Vils, L. (2016). Estudo Bibliométrico: Orientações sobre sua Aplicação. Revista Brasileira de Marketing, 15(2), 246–262. https://doi.org/10.5585/remark.v15i2.3274

Revilla, M., & Ochoa, C. (2017). Ideal and maximum length for a web survey. International Journal of Market Research, 59(5), 557-565. https://doi.org/10.2501/IJMR-2017-039

Roßmann, J., Gummer, T., & Silber, H. (2018). Mitigating satisficing in cognitively demanding grid questions: Evidence from two web-based experiments. Journal of Survey Statistics and Methodology, 6(3), 376-400. https://doi.org/10.1093/jssam/smx020

Rouse, S. V. (2019). Reliability of MTurk data from masters and workers. Journal of Individual Differences. 41(1), 30–36. https://doi.org/10.1027/1614-0001/a000300

Sammut, R., Griscti, O., & Norman, I. J. (2021). Strategies to improve response rates to web surveys: a literature review. International Journal of Nursing Studies, 123, 104058. https://doi.org/10.1016/j.ijnurstu.2021.104058

Saris, W. E., Revilla, M., Krosnick, J. A., & Shaeffer, E. M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61–79.. https://doi.org/10.18148/srm/2010.v4i1.2682

Sauer, P. C., & Seuring, S. (2023). How to conduct systematic literature reviews in management research: a guide in 6 steps and 14 decisions. Review of Managerial Science, 17, 1899–1933. https://doi.org/10.1007/s11846-023-00668-3

Sauermann, H., & Roach, M. (2013). Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features. Research Policy, 42(1), 273-286. https://doi.org/10.1016/j.respol.2012.05.003

Shamon, H., & Berning, C. C. (2020). Attention Check Items and Instructions in Online Surveys with Incentivized and Non-Incentivized Samples: Boon or Bane for Data Quality?. Survey Research Methods, 14(1), 55-77. https://doi.org/10.18148/srm/2020.v14i1.7374

Singer, E., & Ye, C. (2013). The Use and Effects of Incentives in Surveys. Annals of the American Academy of Political and Social Science, 645(1), 112-141. https://doi.org/10.1177/0002716212458082

Smith, S. M., Roster, C. A., Golden, L. L., & Albaum, G. S. (2016). A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples. Journal of Business Research, 69(8), 3139–3148. https://doi.org/10.1016/j.jbusres.2015.12.002

Stoet, G. (2017). PsyToolkit: A Novel Web-Based Method for Running Online Questionnaires and Reaction-Time Experiments. Teaching of Psychology, 44(1), 24–31. https://doi.org/10.1177/0098628316677643

Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020). Got bots? Practical recommendations to protect online survey data from bot attacks. The Quantitative Methods for Psychology, 16(5), 472-481. https://doi.org/10.20982/tqmp.16.5.p472

Struminskaya, B., Weyandt, K., & Bosnjak, M. (2015). The Effects of Questionnaire Completion Using Mobile Devices on Data Quality. Evidence from a Probability-based General Population Panel. Methods, Data, Analyses, 9(2). https://doi.org/10.12758/mda.2015.014

Tourangeau, R. (2003). Cognitive aspects of survey measurement and mismeasurement. International Journal of Public Opinion Research, 15(1), 3–7. https://doi.org/10.1093/ijpor/15.1.3

Tourangeau, R., Couper, M. P., & Conrad, F. (2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368–393. https://doi.org/10.1093/poq/nfh035

Tourangeau, R., Couper, M. P., & Conrad, F. (2007). Color, labels, and interpretive heuristics for response scales. Public Opinion Quarterly, 71(1), 91–112. https://doi.org/10.1093/poq/nfl046

Walzenbach, S. (2019). Hiding Sensitive Topics by Design? An Experiment on the Reduction of Social Desirability Bias in Factorial Surveys. Survey Research Methods, 13(1), 103–121. https://doi.org/10.18148/srm/2019.v1i1.7243

Weber, S. (2021). A step-by-step procedure to implement discrete choice experiments in Qualtrics. Social Science Computer Review, 39(5), 903-921. https://doi.org/10.1177/0894439319885317

Wells, T., Bailey, J. T., & Link, M. W. (2014). Comparison of Smartphone and Online Computer Survey Administration. Social Science Computer Review, 32(2), 238-255. https://doi.org/10.1177/0894439313505829

Zupic, I., & Čater, T. (2015). Bibliometric Methods in Management and Organization. Organizational Research Methods, 18(3), 429-472. https://doi.org/10.1177/1094428114562629

Zwarun, L., & Hall, A. (2014). What’s going on? Age, distraction, and multitasking during online survey taking. Computers in Human Behavior, 41, 236-244. https://doi.org/10.1016/j.chb.2014.09.041